Secure Copy Tools - scp / sftp / rsync & GUI options
This page tells you how to transfer files using secure transfer mechanisms including scp, sftp, rsync, and associated GUI implementations.
SSH supports transferring files over a secure encrypted link. There are two mechanisms provided, Secure Copy (scp) and Secure File Transfer Protocol (sftp).
scpis a scriptable one-off transfer of a file or folder.rsyncis a tool likescpthat allows you to resume interrupted transfers, ensure that folders are identical in two locations, and incrementally sync content.sftpis an interactive session, allowing you to up or download files and folders.
You will typically need to use these tools to download MRI scans, transfer files to/from the WIN or BMRC clusters or collaborator's systems.
In addition to the above command line tools (available on macOS, Linux and recent versions of Windows) there are various GUI options. We recommend Cyberduck on macOS and Windows (OS App Store or https://cyberduck.io/) and FileZilla on Linux.
Using scp
scp is a command line tool (i.e. you enter it to your terminal prompt).
To use it to move a file from location source to destination use:
if source is a folder then you have to add the recursive -r option to copy it and its content.
For example you can move a file myfile.txt from your home folder to a mounted drive called mydata.
Note that the ~ shortcut may be used, and scp can be given relative or absolute paths. The above source input could therefore be achieved using /Users/username/myfile.txt, or ../myfile.txt from a subfolder of your home directory.
source or destination may be local or remote. I.e. you can move files from your local computer to a remote server, from the remote server to your computer, or even remote server to remote server.
We specify the remote location using the syntax
So, to move a file from your local home folder to your home folder on ood/clint, use:
or to move a folder in your ood/clint scratch to your current local folder
Note that if your username matches on your local machine and the server you can omit the username@.
When not to use scp
scp is not resumable, so if you need to transfer a very large file (or a folder with many files) you should probably use rsync. It also can't easily synchronise two folders - it only supports replacing existing files and adding new ones, not removing files from the target that don't exist (anymore) on the source - again use rsync for this.
scp and multi-factor authentication can be irritating as you need to enter your password and second factor every time you transfer a file/folder - if the service supports SSH certificate authentication then this can be avoided, but otherwise use sftp or a GUI app.
Wildcards
scp supports wildcards (*) when specifying source to allow the transfer of many files/folders with similar names, in this case destination must be a folder that they will be copied to. Note that if the source location is remote, the wildcard will need to be escaped (\*) or the whole expression wrapped in quotes, e.g., 'username@sftp.fmrib.ox.ac.uk:~/data/*.nii.gz'.
Other tweaks and encryption
Where your local SSH client and the remote server supports it, you will usually see a significant performance improvement by using the option -o Ciphers=aes128-gcm@openssh.com.
To make this the default for a particular host you can put this in your local ~/.ssh/config file:
N.B. If your data use agreement/DPIA stipulates that data transfers must be encrypted to AES256 standard then switch this out for aes256-gcm@openssh.com - this is somewhat slower.
Using rsync
Using rsync is similar to using scp. However, it allows you to transfer files/folders between two locations in a resumable manner and optionally ensure that the two locations are identical. It uses file metadata to determine what needs to be transferred which means it is much slower to start transferring than scp but for regular mirroring or large transfers, especially over intermittent network connections (such as WiFi) the efficiency gains over scp are significant.
A simple use case would be transferring a folder from your computer to a remote server. In this case the command looks very similar to scp, using the same syntax to specify the remote location:
As with scp , username@ is optional if the username is the same on your computer as the remote service.
The above command can be reversed, like scp, to retrieve a folder from a remote source.
You can also move single files, by specifying a file name on the remote,
or by specifying a folder location: Note that in the last examplerfolder doesn't need to exist, and will be created, as long as it has a trailing slash. In fact, it is generally recommended to add a training slash when dealing with folders to avoid some common rsync gotchas.
Understanding options and constructing rsync commands
You'll have noticed that -avz has been added to commands. These are short flags that change rsyncs behaviour. We'll explain some of these below, but the Rsyncinator website can help you get to grips with rsync and its (optional) tricky syntax to include / exclude files by name / type etc.
The flags we have suggested here do the following:
-a: archive mode. This turns on lots of useful options which make the target file/folder as closely match as possible but does not turn on the deletion options needed for a true mirror.-z: compress during transfer. On slow links with compressible date this can speed up transfers considerably, but it may be counter productive for files such as JPEGs, PNGs, video files, NIFTI-GZ files and other pre-compressed files.-P: Turns on partial mode and show progress. The former keeps partially transferred files if the connection is lost, which can then be resumed later, the latter displays the transfer's progress.
Using sftp
The sftp command provides an interactive file transfer client, although it can be used in a one-off transfer manner to get individual files. Whilst it is more complex to use than scp, it may be more efficient when you have many files to transfer that can't be specified (easily) with wildcards, especially when you are connecting to a system using multi-factor authentication (such as BMRC's cluster).
In addition, some services only support SFTP connections, such as the OxCIN MRI download system when used with Data Access accounts.
The basic sftp command syntax is
wheredestination is of the form username@host:path. As with scp and rsync, username@ is optional if the username is the same on your computer as the remote service. If path is not specified then it will use your remotely configured home folder.
When you have authenticated you will get a prompt that can be used to move around the remote file system; use cd and ls as you would on a UNIX system to change directory and list the contents. You can also interact with your local file system with the lcd and lls.
To upload files/folders use put and to download use get with the filename of the local or remote object respectively.
If you will be copying large volumes of files then you make be able to improve performance by requesting the use of a hardware accelerated encryption mode, do this with -c aes128-gcm@openssh.com
To make this the default for a particular host you can put this in your local ~/.ssh/config file:
N.B. If your data use agreement/DPIA stipulates that data transfers must be encrypted to AES256 standard then switch this out for aes256-gcm@openssh.com - this is somewhat slower.
GUI options
CyberDuck (for macOS and Windows users)
CyberDuck is free software. If your computer is Orchard managed CyberDuck should already have been installed as part of the software bundle included.
Once installed and running, you can open a connection using the Open Connection globe icon. From the drop-down choose SFTP (Secure File Transfer Protocol) and then enter the host name into the box provided. Unless told otherwise by instructions for the host you are connecting to leave the port set to 22.
You may enter your username and password or leave these blank to be prompted when connecting. Please do not store your password in the Keychain (on macOS) - we recommend using a Password Manager instead.
Click Connect to connect to the remote server. You can then navigate the remote folder and drag/drop files to/from as required.
If this is the first time you have connected to a particular server you will be given the 'fingerprint' of the server - if this is a OxCIN server you can check this on our SSH page. Tick the box to always accept this fingerprint - should it change on the server, you will be notified and connections will not be attempted.
Multi-Factor Authentication (MFA) with CyberDuck
If the remote server requires MFA then the default settings for CyberDuck will mean that for each transfer you request you will need to enter your credentials afresh. To avoid this, change CyberDuck's preferences to use a persistent connection. Open the Preferences window then in Transfers > General, change Transfer Files to Use browser connection.
FileZilla (For linux users)
Linux users should use FileZilla. It should be a simple install from your OS's software centre.
When you launch FileZilla you can connect simply by filling out the Quickconnect section at the top, hostname, username and password and port 22 (for SFTP). The first icon in the tool bar allows you to save connections for multiple servers.
Never agree to store your password in FileZilla's store - consider using a Password Manager instead if you are unable to remember your password.
The first time you connect to a server you will be given a prompt advising you that the host key for this server is unknown - for OxCIN servers you can check this on our SSH page. Tick the 'Always trust this host...' box if it is correct and then click OK to continue. If the key were to change you will be warned and connections will not be made. Other services should publicise their server fingerprints.
Once connected you will have a view of your local files on the left and the remote folder on the right, drag/drop between these sections to transfer files.
MFA with FileZilla
To avoid the need to continually enter your password and second factor when transferring multiple files during a session, use the Site Manager (first icon on the tool bar) to create a new site. Give it a memorable name, e.g. BMRC, choose the SFTP - SSH File Transfer Protocol option, fill in the host field then in the Transfer Settings tab tick the Limit number of simultaneous connections and ensure that the maximum number of connections is set to 1.
Leave the Logon Type set to Ask for password - consider using a Password Manager instead if you are unable to remember your password.
Click OK to save.
To open a saved connection, visit the Site Manager, select the site in the tree on the left and click the Connect button.