You are here

Migration Process

Compute Canada has begun one of the biggest advanced research computing renewals in Canada’s history. Funding from the Canada Foundation for Innovation (CFI) and provincial partners is being used to replace several ageing systems with four new national systems that will consolidate resources and centralize services. For more information on the national migration process and the latest update to the technology refresh program, please visit the Compute Canada website

This page provides details specifically about migrating users and data off WestGrid’s legacy systems including any action WestGrid users need to take.

Thank you in advance for your patience during this migration period. We are working hard to limit the extent of disruptions or impacts to your research during this transition. If you have questions or concerns about any aspect of this migration process, please contact support@westgrid.ca

Table of Contents:

WestGrid Systems Defunding Schedule

*** Note“Defunded” means that the system is no longer funded, operated or maintained as part of Compute Canada’s national cyberinfrastructure platform, which is funded through the Canada Foundation for Innovation’s (CFI) Major Science Initiatives (MSI) program. After this time the system is no longer available for use by Compute Canada or WestGrid users.

Once a system has been defunded, local institutions may choose to keep a system running, at their own cost and discretion. This may result in usage limited to a certain group of users (e.g. only those from the local institution).

Users are encouraged to contact the Site Lead of the defunded system for further information.***

 

Legacy Systems to be Defunded in 2017

System(s) Institution / Location Migration Period Defunding Date Data Deletion Date ACTION REQUIRED...
Silo University of Saskatchewan n/a Mar 31, 2017 n/a Refer to the Compute Canada Documentation Wiki Silo Migration Guide.
Hungabee / Jasper University of Alberta Mar 1- 15, 2017 Mar 31, 2017 April 1, 2017

Please start migrating any data off these systems.

Refer to the Compute Canada Documentation Wiki WestGrid Migration Guide.

Note: Limited warranty support after January 1, 2017. See below for details.

Hermes / Nestor University of Victoria Mar 1 - 31, 2017 April 15, 2017 June 1, 2017

Please start migrating any data off these systems.

Refer to the Compute Canada Documentation Wiki WestGrid Migration Guide.

Breezy / Lattice University of Calgary Mar 1 - 31, 2017 April 15, 2017 June 1, 2017

Please start migrating any data off these systems.

Refer to the Compute Canada Documentation Wiki WestGrid Migration Guide.


Users are asked to move their own data to a new system during the Migration Period listed above and BEFORE the Defunding Date.

Any remaining data WILL BE DELETED (refer to the Data Deletion Dates in the previous table). Please note that WestGrid will not retain any long term back-up copies of user data. Users should ensure they take the appropriate steps to comply with any data management requirements their institution may require.

Legacy Systems to be Defunded in 2018

System(s) Institution / Location Defunding Date What To Do...
Parallel University of Calgary Mar 31, 2018 Wait for further instruction from the WestGrid Support Team. If you have questions, email support@westgrid.ca.
Orcinus University of British Columbia Mar 31, 2018 Wait for further instruction from the WestGrid Support Team. If you have questions, email support@westgrid.ca.

Note: Limited warranty support after January 1, 2017. See below for details.
 Grex University of Manitoba Mar 31, 2018 Wait for further instruction from the WestGrid Support Team. If you have questions, email support@westgrid.ca
 Bugaboo Simon Fraser University Mar 31, 2018 Wait for further instruction from the WestGrid Support Team. If you have questions, email support@westgrid.ca

 

Data Retention Policy for Defunded Systems

  • The policy for all systems which will be defunded is to notify users (via any/all of the following methods: newsletter, email, website, MOTD, user docs wiki) at least 90 days before the date data may be deleted
  • Data deletion dates will be posted for all systems which will be defunded.
  • It is the responsibility of the host institution to determine which user data will be deleted and then purge files as needed.
  • It is the host institution’s responsibility to develop and communicate any future policies on system use/access, account management, data retention policies, etc. to users.

IMPORTANT: Please note that WestGrid will not retain any long term back-up copies of user data. Users should ensure they take the appropriate steps to comply with any data management requirements their institution or project may require.

Decreased Performance And Limited Support

Please note that some legacy systems may be operating under limited vendor support and may suffer from decreased reliability due to increased downtime required for maintenance and repairs. Possible risks include lost jobs that will need to be resubmitted and less resources available to users over time if the nodes can not be restored.

Jasper / Hungabee

As of January 01, 2017, hardware support for these systems is limited to critical components only. No new (or replacement) compute nodes will be added which will likely result in less available compute nodes and cores over time as components die. We also expect the reliability of these resources to decline with downtime increasing due to increased need for maintenance and repairs. 

Orcinus

As of January 01, 2017 the Orcinus storage system is not under vendor support. Due to the redundant nature of the storage controllers we feel that this is an acceptable risk. Orcinus will be backed up to the new storage system, the National Data Cyberinfrastructure  (NDC) located at SFU and Waterloo. There may be some lack of continuity in backups as we transition between legacy and new tape systems, so we recommend users keep their own copies of critical data.  

How to Migrate your Data

Users are asked to move their own data to a new system BEFORE the Defunded Dates (see the tables above).

  1. Please DELETE any unneeded files.
  2. Please email support@westgrid.ca to NOTIFY US WHEN you have moved your data. 

We recommend users move their data well in advance of the Defunded Date to avoid network bottlenecks with file transfers, etc.

If you are responsible for your own data migration, please refer to the General directives for migration on the Compute Canada Documentation Wiki for detailed information on the following:

  • How to transfer files to other systems
  • How to use Globus to transfer files
  • Details about the new national GP (General Purpose) systems

Note that the migration of certain legacy systems like Silo is done by WestGrid / Compute Canada staff. 

Where to Migrate to?

The 2017/2018 Resource allocation year (April 1, 2017 to March 31, 2018) will be a transition year with a mix of legacy and new systems available for use. The table below details the WestGrid legacy systems which will be available to all WestGrid users, the storage quotas and intended purpose of the systems. 

System

Default Storage

Backup policy

Type of jobs

Bugaboo

/home: 300 GB per user, 1M files


/global/scratch: 1 TB per user, 1M files/user


/scratch: depends on compute node

Daily backup



No backup


No backup. Files removed after job completion

Bugaboo is intended for jobs that require access to large storage systems (terabytes of data, for example).

More generally, the Bugaboo system can be used as a general purpose system appropriate for serial and parallel jobs and jobs that require long walltimes. There are a few nodes available that provide 48000 MB of memory - most nodes have either 16000 MB or 24000 MB.

Orcinus

/home: 250 GB per user, 250,000 files


/global/scratch: 500 GB per user, 200,000 files

Backed up



No backup

Orcinus is intended for both serial and parallel applications.

Grex

/home: 30 GB per user, 500,000 files


/global/scratch: 800 GB per GROUP, 1M files/user

Daily backup



No backup. Long-term data retention not guaranteed.

Grex is primarily intended for applications that can take advantage of non-blocking low latency InfiniBand networks and large memory per compute node. It also hosts the WestGrid-wide license for Gaussian computational chemistry software. Grex is a general-use system so no special access requests required for registered WestGrid users.

Parallel

/home: 50GB, 200,00 file limit per directory




/global/scratch: 450 GB, 200,000 files; email support to increase quota

Users are responsible for their own backups.


Files older than 30 days are subject to deletion.

Parallel is intended for multi-node parallel applications that can run in a relatively short time (less than 3 days) and can take advantage of its InfiniBand interconnect or special GPU-based nodes. It can also be used for applications that have license restrictions that prevent them from being run elsewhere.

 

More information on suggested systems to migrate to including information about the new national systems can be found on the Compute Canada Documentation Wiki

Getting Started on the New Systems

Compute Canada's new national systems should be available in Spring 2017. Descriptions of the new systems can be found on the New Systems page of the Compute Canada Documentation Wiki. 

As a rule-of-thumb those users located in the West (Saskatchewan to BC) should choose Cedar-GP2, and the rest should choose Graham-GP3. This has no particular technical justification but does give at least some guidance for the choice. For Cedar-GP2 and Graham-GP3, the parallel filesystems, interconnects and external persistent storage is the same with a slightly different mix of compute nodes.

Note: WestGrid users will need to use their Compute Canada username and password to login to the new systems (the username and password used to log in to ccdb.computecanada.ca). See User Accounts and Groups on the Compute Canada Documentation Wiki for more information.

FAQs

For Frequently Asked Questions (FAQs) about migration, the national systems, and other Compute Canada services, visit the FAQ page on the Compute Canada Documentation Wiki.

Support & Other Links

Please email support@westgrid.ca for general help and questions at any point during the migration process, or refer to the following links for more tools, tips and support related to migration: