Introduction

Please see the Running Jobs page for a general introduction to the batch queuing and scheduling system used at WestGrid sites. Additional site-specific information for the Matrix system is given below.

File Systems and Storage

Users home directories are in the /home/users file system. A quota is enforced, requiring users to limit usage to less than 10 GB. The file /home/users/DU gives a summary of disk usage in the /home/users file system. Although there may be some ad hoc backups of /home/users, this is not guaranteed. Users are responsible for backing up their own files.

In addition, there is a larger file system, /scratch, for storage of large files associated with running jobs. A subdirectory /scrach/<username> is available for each user. A quota of 350 GB per user is enforced for the /scratch directory (to prevent runaway jobs from using up all the space), although it is suggested that usage be limited to less than 100 GB. If you need more than that to support running jobs, please be particularly vigilant about cleaning up as jobs complete. /scratch/DU gives a summary of disk usage in the /scratch file system.

If you need to keep files for more than 30 days, please move them out of /scratch to the WestGrid storage facility or your own system. The gcp (/usr/apps/bin/gcp) command can be used to efficiently transfer files to the storage facility. See the Gridstore QuickStart Guide for more information.

Jobs that produce significant output should be started from your subdirectory under /scratch and not from your home directory.

Batch Jobs and Usage Guidelines

As described on the main Running Jobs page, batch jobs should be submitted with qsub, can be monitored with qstat or showq and deleted with qdel. Sample jobs scripts are available on the Matrix Programming page. In particular for MPI-based parallel jobs, you must submit with a Matrix-specific job script, so, do not just copy a script from another WestGrid system.

Some points to keep in mind when submitting jobs on Matrix:

• Matrix is intended only for parallel jobs that require a high-performance interconnect. Except for minor testing, debugging or benchmarking, it is expected that only parallel jobs requesting multiple nodes will be submitted.
• As noted above, jobs that produce significant output should be started from your subdirectory under /scratch and not from your home directory.
• The maximum walltime limit is 72 hours.
• The maximum number of processors that can be used at any one time by a single user is 96.
• There are two processors on each node so that maximum processors per node (ppn) is 2. In order to facilitate scheduling, please use -l nodes=...:ppn=2 for all jobs unless memory requirements dictate otherwise. See the main Running Jobs page for more information about how to specify ppn.
• The Matrix compute nodes each have slightly less than 2 GB of RAM available. If using the -l mem=... resource request, use 2000MB per node rather than 2GB. So, for example, to request essentially all the memory on 16 nodes, use -l mem=32000MB.
   
  Note affecting users requesting 64 processors: The batch scheduling system converts resource requests, of which the memory request is a part, into processor equivalents. After a system upgrade in January 2009, the number of equivalent processors calculated in this way increased slightly for a given amount of memory requested.  This led to an error for jobs requesting 64 processors with -l nodes=32:ppn=2,mem=64000MB .  A workaround for this is to request slightly less memory: -l nodes=32:ppn=2,mem=62000MB. This will keep the number of equivalent processors less than the limit of 64 per user mentioned above.

Interactive Sessions

Limited interactive work can be done on the Matrix login node. For more extended interactive sessions for debugging or visualizaiton, submit an "interactive batch job" with a command like

qsub -I -X -l walltime=00:30:00

as described on the main Running Jobs page.

Although there are no compute nodes specifically reserved for interactive work, two nodes do have a maximum walltime limit of 3 hours and are usually available without a long wait.

Updated 2010-05-18 - Maximum total number of simultaneous processors per user raised to 96 from 64.