If you need to move large files between WestGrid sites, globus-url-copy is an order of magnitude faster than scp. Before you can use globus-url-copy, you need to initialize a proxy.
In this example, we will move the file large.dat from our home directory on bugaboo to the /tmp directory on our storage site, silo.
globus-url-copy file:/home/users/juser/large.dat gsiftp://silo/tmp/large.dat
There are options we can use to boost the speed of copying:
- The -tcp-bs 4194304 option requests a buffer size of 4MB, which we have found is a good size for fast transfers at WestGrid.
- Another pair of options which has proven successful at WestGrid is -tcp-bs 2097152 -p 2. This runs two parallel transfer streams using 2MB buffers.
Putting this all together:
globus-url-copy -tcp-bs 2097152 -p 2 file:/home/users/juser/large.dat gsiftp://silo/tmp/large.dat
The globus-url-copy command can currently copy only files and not directories. If you need to move a directory, you might can use the tar or gzip commands to wrap it all up into a file.
For more detail, use globus-url-copy -help.
See also gcp