You are here

Hermes/Nestor

***NOTE: The Hermes and Nestor systems will be “defunded” on June 1, 2017. Please visit the Migration Process page for more information.

The Hermes and Nestor system consists of two clusters that share infrastructure such as resource management, job scheduling, networked storage, and service and interactive nodes. Hermes is a capacity cluster geared towards serial jobs. The Hermes cluster consists of 84 nodes having 8 cores each and 120 nodes with 12 cores each, which gives a total of 2112 cores. Nestor is a capability cluster consisting of 288 nodes (2304 cores) geared towards large parallel jobs.

This system is also documented as part of the University of Victoria's Research Computing Facility.

We recently virtualized all the hermes nodes and move them to the cloud. The new nodes have 28 cores, Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz. This change should be transparent to the user and the jobs should work as they used to (the performance should be not that much impacted and actually may be better for some jobs).

hermes.westgrid.ca and nestor.westgrid.ca are aliases for a common pool of head nodes named litaiNN.westgrid.ca. The choice of destination hostname when connecting to the system has no bearing on what kind of jobs may be run.  One may log in to hermes.westgrid.ca and submit a parallel job to the Nestor cluster. See the Processor specification section below for more information on how job submissions are handled.

Email from batch jobs is not currently supported. So, do not use the TORQUE -M and -m directives in your batch job scripts on Hermes and Nestor.

The spinning disk storage is a DDN system structured into 45 tiers. Each tier provides 16 TB of usable storage. The amount left (180 TB) is used for RAID. The following table summarizes the current usage of different tiers:

Number of Tiers Amount of usable TB Usage
12 192 GPFS
29 464 dCache (for ATLAS project)
1 16 NFS (for ATLAS project)
1 16 TSM for system backup
2 32 Spare

 

The total amount of usable spinning disk is 720 TB including the file system overhead. 180TB of disk space is managed by four storage nodes and shared amongst multiple uses, including home directories, software, and scratch space.

Disk usage is monitored but automatic disk quotas (i.e. using the "quota" command) are not implemented, nor are there current plans to implement this mechanism. Quotas are given below.

Key file spaces, their intended uses, backup policies and quotas are as follows:

/home

  • /home/username is your home directory (assigned to the HOME environment variable).
  • Only essential data should be stored here, such as source code and processed results.
  • Backed up nightly; most recent backup once active copy deleted is stored for 180 days.
  • Quota: 300GB per user.

/global/scratch

  • /global/scratch/username is your scratch directory.
  • This is your work area for jobs. Please use this for data sets and job processing.
  • This file area is not backed up.
  • Quota: 1TB per user.

/global/software

  • This is where most software of user interest is installed, such as applications, analysis frameworks and support libraries.
  • A list of such software is available below, but for a current up-to-date list please use ls /global/software. Most of these software can be nicely accessed using modules.
  • This file area is backed up nightly.

/scratch: Local scratch space on the nodes

The first 84 Hermes nodes and all Nestor nodes have a 250GB drive with about 225GB available for local, non-persistent scratch space for the lifetime of the job. This is roughly 28GB of scratch space per core.

The new 120 Hermes nodes have 500GB storage space (~433GB is used for scratch).

The scratch space can be accessed via the environment variable $TMPDIR.

Using scratch space in your job

The usual usage of the node's local scratch space is to first copy the necessary files to $TMPDIR, perform processing, and copy the results back from $TMPDIR to your home or global scratch space, as appropriate.

Storage Information on Hermes/Nestor

Directory path Size Quota Command to check quota Purpose Backup Policy
/home 300GB/user and 200000 files

Only essential data should be stored here, such as source code and processed results.

Backed up nightly; most recent backup once active copy deleted is stored for 180 days.

/global/scratch 1TB/user, 500000 files

This is your work area for jobs. Please use this for data sets and job processing. 

No backup

/global/software

This is where most software of user interest is installed, such as applications, analysis frameworks and support libraries.

For a current up-to-date list please use ls /global/software. Most of these software can be nicely accessed using modules.

This file area is backed up nightly.

/scratch: Local scratch space on the nodes

The first 84 Hermes nodes and all Nestor nodes have a 250GB drive with about 225GB available for local, non-persistent scratch space for the lifetime of the job. This is roughly 28GB of scratch space per core.

The new 120 Hermes nodes have 500GB storage space (~433GB is used for scratch).

The scratch space can be accessed via the envrionment variable $TMPDIR.

Program Information on Hermes/Nestor

Wall time specification

Wall time is the amount of real time in which a job runs, regardless of the amount of CPU used or other factors. In other words, the amount of time recorded on a wall clock.

By default, jobs have a wall time of one minute. This encourages users to specify a more realistic wall time. Typically users estimate and multiply by three. To specify a wall time for a job, include the following directive at the top of the submission script (this example is for a 24-hour job):

#PBS -l walltime=24:00:00

Wall times enable prioritization and queuing based on the length of time resources will be consumed, and to some extent may be used by users to predict when their queued jobs may run.

The maximum walltime on Nestor and Hermes is 72 hours (3 days). The maximum number of jobs that a user can have in the queue is 5000. As such, the maximum size of an array job is also 5000. For more information about the scheduling policies on Nestor and Hermes please check the Nestor/Hermes Job Scheduling page.

Processor specification

Both Hermes and Nestor clusters are accessed via the same resource manager with two main queues: hermes for the Hermes cluster and nestor for the Nestor cluster. The qsub wrapper will figure out the right queue based on the job requirements (by looking at the #PBS options inside the PBS script) if the user did not specify the queue.

One may request a specific number of processors with the procs directive. (Processors chosen by the scheduler may be on any node, so, this is not appropriate for programs using OpenMP or other parallel programming approaches that are restricted to a single node). In this example, eight processors are requested:

#PBS -l procs=8

If you are submitting a parallel job to Hermes, then the maximum number of cores  that can be requested is 12 and they should be specified using -l nodes=1:ppn=X (where X is less than or equal to 12) instead of -l procs=X.  If you want to use more than 12 cores with the procs resource request, you must also explicitly request the nestor queue by using -q nestor.  For example:

#PBS -q nestor
#PBS -l procs=24

One may also request multiple processors on a single node:

#PBS -l nodes=1:ppn=8

Finally, one may also request multiple processors on multiple nodes:

#PBS -l nodes=4:ppn=8

Jobs requesting 8 cores or less should run on hermes, and the others should run on nestor.

Note that with parallel programs that use the Message Passing Interface (MPI), it is better to use procs as it is quicker to schedule.

Memory specification

Each node has 24GB of memory, of which 1-2GB is used for the OS, depending on the image used, as well as the GPFS cache. This leaves roughly 22GB of memory for jobs. The default amount of memory per job is 1024MB. To specify more, a resource directive like the following may be used (this example is of course for 2GB):

#PBS -l mem=2048mb

The mem parameter is the total memory limit for a job. For a parallel job, the pmem parameter can be used to specify a per-process memory requirement. For example:

#PBS -l procs=16,pmem=2gb

This example requests 16 processors with 2GB of memory per process (this gives 32GB of total memory).

2015-04-06 - Noted that email from batch jobs is not supported.
2015-11-04 - Clarified that -q nestor is needed for procs>12.