You are here

Home » Resources » WestGrid Computing Facilities

WestGrid Computing Facilities

Introduction

The WestGrid computing facilities are distributed among several resource provider sites, with some specialization at each site. WestGrid is connected by high-performance networks so that users can access the system which best fits their needs, regardless of where it is physically located.

The systems are for high performance computing, so they are something beyond what you would find on a desktop. WestGrid provides several types of computing systems, since different users' programs will run best on different kinds of hardware.  We have clusters, clusters with fast interconnect, and shared memory systems. 

Which system should you use?

Use the system which best fits your needs, not necessarily the one closest to you. 

See the QuickStart Guide for New Users introduction to choosing the most appropriate system. For more detailed technical information about the differences between the WestGrid systems, read the information below and in the linked pages for each system.

National Resources

WestGrid's computing facilities are part of Compute Canada's national platform of Advanced Research Computing resources.  

New National Sites

Four new national sites will offer expandable and modern data centres with highly qualified personnel. Operations of the new systems will be delivered through national teams, drawing upon regional and local expertise distributed across Canada. 

Location

System Name
University of Victoria Arbutus / GP1 / Cloud-West
Simon Fraser University Cedar / GP2
University of Waterloo Graham / GP3
University of Toronto Niagara / LP (Large Parallel)

SFU & Waterloo

 National Data Cyberinfrastructure (NDC -Storage)

 

Please refer to the CC User Documentation wiki for more information on national systems and services.

Migration to the New National Systems

Compute Canada has begun one of the biggest advanced research computing renewals in Canada’s history to replace several ageing systems with four new national systems. For more information on the national migration process and the latest update to the technology refresh program, please visit the Compute Canada website. 

All current WestGrid systems were installed 2010-2012 and will be defunded over the next two years. For details about this process, as well as key information and instructions for WestGrid users, visit our Migration Process page.

WestGrid Systems

System(s) Site Cores Type Details
Breezy University of Calgary 384 Shared memory

NOTE: This system will be “defunded” in August 2017. Please visit the Migration Process page for more information.

  • Appro
  • 24 nodes: quad-socket, 6-core AMD 2.4GHz nodes
  • 256 GB per node
  • Infiniband 4X QDR
  • Dell FluidFS file system
Bugaboo Simon Fraser University 4584 Storage, Cluster with fast interconnect
  • Dell
  • 160 nodes: 8 cores, Xeon X5430 with 16 GB/node = 1,280 cores (Infiniband, 2:1 blocking)
  • 254 nodes: 12 cores, Xeon X5650 (212 nodes with 24 GB/node, 32 nodes with 48 GB/node) = 3,048 cores (Infiniband, 2:1 blocking)
CC-Cloud University of Victoria 7640 OpenStack Cloud

Visit the CC-Cloud Resources page on the Compute Canada User Wiki for full system details.
Grex University of Manitoba 3792 Storage, Cluster with fast interconnect
  • SGI Altix XE 1300
  • 316 compute nodes
  • 2 x 6core Intel Xeon X5650 2.66 MHz processors per node
  • 24 nodes have 96 GB, 292 nodes have 48 GB
  • Infiniband 4X QDR
Hermes/Nestor University of Victoria 4416 Storage, Cluster with fast interconnect

NOTE: The Hermes and Nestor systems will be “defunded” on June 1, 2017. Please visit the Migration Process page for more information.

Hermes

  • Original nodes: 84 x 8 core, IBM iDataplex X5550 2.67 GHz, 24 GB/node, 2 x GigE interconnects
  • Newer nodes: 120 x 12 core, Dell C6100 servers, 2.66 GHz X5650 cores with 24 GB/node, QDR IB 10:1 blocking
  • GPFS 1.2 PB for home, scratch (shared with nestor)

Nestor

  • 288 x 8 core/node, IBM iDataplex X5550 2.67 GHz, 24 GB/Node
  • QDR IB nonblocking
  • GPFS 1.2 PB for home and scratch (shared with hermes)
Hungabee University of Alberta 2048 Shared memory

NOTE: This system will be “defunded” in Fall 2017. Please visit the Migration Process page for more information.

  • Special Request Only
  • SGI UV1000, NUMA Shared-memory
  • 2048 Intel Xeon E7 cores
  • 16 TB total (shared) memory
  • NFS: 2 x SGI IS5000 storage arrays
    • 8 x fibrechannel direct to the UV1000. (short term storage)
    • 50 TB
  • Lustre: 1 x SGI IS16000 array 355 TB. (medium term storage)
    • Available to BOTH Hungabee and Jasper through QDR IB
Jasper University of Alberta 4160 Cluster with fast interconnect

NOTE: This system will be “defunded” in Fall 2017. Please visit the Migration Process page for more information.

  • SGI Altix XE, 400 nodes, 4160 cores and 8320 GB of memory
    • 204 Xeon X5675 nodes - 12 cores (2 x 6), 24 GB, 40 Gbit/sec 1:1 Infiniband interconnect
    • 36 Xeon X5675 nodes - 12 cores (2 x 6), 48 GB, 40 Gbit/sec 1:1 Infiniband interconnect
    • 160 Xeon L5420 nodes - 8 cores (2 x 4), 16 GB, 20 Gbit/sec 2:1 Infiniband interconnect
  • Lustre parallel distributed filesystem, 356 TB - shared with all nodes via Infiniband
Lattice University of Calgary 4096 Storage, Cluster with fast interconnect

NOTE: This system will be “defunded” in August 2017. Please visit the Migration Process page for more information.

  • 512 x 8-core nodes.
    • Intel Xeon L5520 quad core 2.27 GHz
    • 12 GB/node
  • QDR IB (2:1 blocking factor)
Orcinus University of British Columbia 9600 Storage, Cluster with fast interconnect
  • Phase 1: 384 nodes, 3072 cores
    • 8 cores/node
    • Xeon E5450 3.0GHz
    • 16 GB Ram
    • DDR IB
  • Phase 2: 554 nodes, 6528 cores
    • 12 cores/node
    • Xeon X5650 2.66 GHz
    • QDR IB
  • IB with 2:1 blocking factor
  • Phase 1 and Phase 2 share filesystems but otherwise run as separate systems
ownCloud Simon Fraser University Storage
Parallel University of Calgary 7056 Storage, Cluster with fast interconnect, Visualization
  • HP ProLiant SL390
  • 528 x 12 core nodes
    • Intel E5649 (6 core) 2.53 GHz
  • 60 special 12 core nodes with GPU
    • NVidia Tesla M2070s (5.5 GB ram and Compute Capability 2)
  • IB QDR (2:1 blocking factor to reduce costs)
  • Global scratch shared between breezy, lattice and parallel
WestGrid portal None

List of WestGrid facilties by general type

  • Storage
    • USask Storage Facility (Silo) -- the primary storage site
    • UVic Storage Facility and SFU Storage facility -- for use in special cases where there is a need for large storage close to the compute nodes
  • Shared memory
    • Hungabee
  • Cluster
    • Hermes, Breezy (large memory)
  • Cluster with fast interconnect
    • Bugaboo, Grex, Jasper, Lattice, Nestor, Orcinus, Parallel
  • Visualization
    • Parallel has special nodes with Graphics Processing Units (GPUs).

Retired WestGrid Systems

Some older WestGrid systems have been removed from general service, typically being replaced with more energy-efficient machines with more capability.

Machine name Period of Service Description

Silo

 Dec. 2008 - Mar. 2017

Silo was the primary storage facility at WestGrid with over 3.15 PB (3150 TB) of spinning disk. It was an archival facility, which was backed up. There were two main login servers: Silo and Hopper. Silo and Hopper shared filesystems.

Disk storage: total 4.2 PB raw, 3.15 PB usable
600 x 1TB SATA drives
1800 x 2TB SATA drives
RAID 6
2 pairs of Dual IBM/DDN DSC9900 Controllers

Tape System: IBM LTO 3584 tape library
6 frames capable of holding 6000 LTO tapes
6 LTO4 drives
6 LTO5 drives
1780 x LTO4 tapes (averaging >1TB/tape with compression)
1400 LTO5 tapes (averaging 1.5TB/tape with compression)

Backup Software: IBM Tivoli Storage Manager (TSM)