Efficient file transfers with globus-url-copy

How can I move my files quickly between WestGrid centres?

If you need to move large files between WestGrid sites, globus-url-copy is an order of magnitude faster than scp. Before you can use globus-url-copy, you need to initialize a proxy.

In this example, we will move the file large.dat from our home directory on lattice to the /tmp directory on our SFU storage site, gridstore.

There are options we can use to boost the speed of copying;

• The -tcp-bs 4194304 option requests a buffer size of 4MB, which we have found is a good size for fast transfers at WestGrid.
• Another pair of options which has proven successful at WestGrid is -tcp-bs 2097152 -p 2. This runs two parallel transfer streams using 2MB buffers.

The globus-url-copy command can currently copy only files and not directories. If you need to move a directory, you might can use the tar or gzip commands to wrap it all up into a file.

For more detail, use globus-url-copy -help.

See also gcp