You are here

Efficient file transfers with globus-url-copy

If you need to move large files between WestGrid sites, globus-url-copy is an order of magnitude faster than scp. Before you can use globus-url-copy, you need to initialize a proxy.

In this example, we will move the file large.dat from our home directory on bugaboo to the /tmp directory on our storage site, silo.

globus-url-copy file:/home/users/juser/large.dat gsiftp://silo/tmp/large.dat

There are options we can use to boost the speed of copying:

  • The -tcp-bs 4194304 option requests a buffer size of 4MB, which we have found is a good size for fast transfers at WestGrid.
  • Another pair of options which has proven successful at WestGrid is -tcp-bs 2097152 -p 2. This runs two parallel transfer streams using 2MB buffers.

Putting this all together:

globus-url-copy -tcp-bs 2097152 -p 2 file:/home/users/juser/large.dat gsiftp://silo/tmp/large.dat

The globus-url-copy command can currently copy only files and not directories. If you need to move a directory, you might can use the tar or gzip commands to wrap it all up into a file.

For more detail, use globus-url-copy -help.

See also gcp