Data Transfer Methods

SFTP (Secure File Transfer Protocol)

SFTP clients are interactive file transfer programs, similar to FTP, which perform all operations over an encrypted transport.

A variety of graphical SFTP clients are available for different OSes:

You can also SFTP using the command line.

The general syntax to copy a file to a remote server is:

$ echo "put <source_file_path>" | sftp <username>@<remote_host>:<destination_path>

For instance, the following command will copy the file named foo from your local machine to your home directory on Nero:

$ echo "put foo" | sftp <sunetid>@nero.compute.stanford.edu:

Note the : character, that separates the hostname from the destination path. Here, the destination path is empty, which will instruct sftp to copy the file in your home directory.

You can also copy foo under a different name, or to another directory, with the following commands:

$ echo "put foo" | sftp <sunetid>@nero.compute.stanford.edu:/share/pi/path/here

And finally, sftp also support recursive copying of directories, with the -r option:

$ echo "put -r dir/" | sftp <sunetid>@nero.compute.stanford.edu:dir/

rsync

If you have complex hierarchies of files to transfer, or if you need to synchronize a set of files and directories between your local machine and Nero, rsync will be the best tool for the job. It will efficiently transfer and synchronize files across systems, by checking the timestamp and size of files. Which means that it won’t re-transfer files that have not changed since the last transfer, and will complete faster. Recommended rsync command, with the following options, -a archive option (syncs recursively and preserves symbolic links, special and device files, modification times, group, owner, and permissions), -v verbose and -P show progress:

rsync -avP ~/data/ <sunetid>@nero.compute.stanford.edu:data/

Note the slash (/) at the end of the directories name, which is important to instruct rsync to synchronize the whole directories.

For more information about using the rsync, you can refer to this tutorial or for more details and examples.

gsutil rsync (Google Cloud Bucket to Nero)

gsutil is a tool within the Google SDK used to perform an rsync command that transfers files from a Google bucket to Nero. gsutil rsync makes the contents under the destination (Nero path) match the source (Google Bucket), with built-in error-checking and retry attempts.

This assumes you have already done “gcloud auth login” once prior to running the following.

From Nero (ssh or Jupyter Terminal):

$ screen -S google
$ module load google
$ module load anaconda/2
# dry run with the "-n" flag shows you what it would rsync, remove this flag to perform the actual transfer
$ gsutil -m rsync -r -n gs://bucket-name-here/ /share/pi/$PISUNET/path/to/save

Running this command in a screen allows you to let the transfer run in the background, where you can reconnect (with “screen -r” or “screen -rd”) to Nero and check on it later.

More information on the gsutil rsync command can be found on the Google rsync page.