System Notices
Western ROADM Maintenance at Calgary End
TICKET INFORMATION:
Subject: Western ROADM Maintenance at Calgary End
Category: Scheduled maintenance
Ticket ID: 20100708-007
Start Time: 2010-07-13 22:00 EDT (2010-07-14 02:00 UTC)
End Time: 2010-07-13 23:59 EDT (2010-07-14 03:59 UTC)
TICKET HISTORY:
== Updated: Jun on 2010-07-14 02:50 EDT(2010-07-14 06:50 UTC) ==
The maintenance was completed. All lightpathes are
restored and are in service.
== Created: Malcolm on 2010-07-08 15:09 EDT(2010-07-08 19:09 UTC) ==
At the date and time listed, fiber provider will be doing some fiber
cleanup at their colo location in Calgary. They will be cutting over our
dark fiber to a more direct cross connect. This is not expected to last
longer than 30 minutes but the entire window has been reserved.
The following wavelengths will be affected:
VNCN1-CLGR2-01(CANARIE)
VNCN1-CLGR2-02(CANARIE)
STTL1-CLGR2-01(CANARIE)
VNCN1-CLGR2-03(BCnet)
VNCN1-CLGR2-04(WestGrid)
The following lightpaths will be affected:
10313LP02-GreenStarNet-CLCG2-MTRL2
Neptune300 VCTR-SASK
TRIUMF UBC - McGill
TRIUMF Van-Tor
TRIUMF UBC - UofA
TRIUMF - CERN 5G
RDC VNCV1-CLGR2
NRNet VCTR - SASK
NRNet VCTR - RGNA
NRNet VCTR - EDMN
NRNet VCTR - OTWA
ECONET Van - Mon
ECONET VCTR2-EDMN1
CANARIE NOC
Operations and Engineering
Email: eng@canarie.ca
Weekdays: 08:00-17:00 EDT(UTC-5)
+1.613.944.5612
7/24 pager: +1.613.944.5611
http://www.canarie.ca/canet4/
July 13, Bugaboo /home filesystem problem resolved
===========================================================================
July 13, 16:00 - /home filesystem
We have upgraded Lustre file server software in order to fix "No space
left on device" error. We hope this problem is resolved. Please report
any issue you have to support@westgrid.ca
July 10, Bugaboo /home filesystem problems
We continue to have problems with the /home filesystem on the Bugaboo cluster that become apparent through spurious error messages "Cannot write: No space left on device".
This is due to a bug in the filesystem software. We are in contact with the vendor, but no fix is available so far. The vendor suspects that this bug is triggered by certain jobs running on the cluster, but we have not been able to isolate those jobs. In the meantime we know how to fix this problem in the short term (i.e., until the systems runs into this bug again). Thus, continue to email support@westgrid.ca when you run into this problem.
The /global/scratch file system is not affected by this bug so far. Thus, using /global/scratch instead of /home presents a workaround for this problem. A word of caution though: there is no reason to assume that the same bug could not be triggered on /global/scratch as well. The fact that we have not seen the problem in /global/scratch probably just indicates that the jobs that trigger the bug use the /home filesystem and not /global/scratch.
This is due to a bug in the filesystem software. We are in contact with the vendor, but no fix is available so far. The vendor suspects that this bug is triggered by certain jobs running on the cluster, but we have not been able to isolate those jobs. In the meantime we know how to fix this problem in the short term (i.e., until the systems runs into this bug again). Thus, continue to email support@westgrid.ca when you run into this problem.
The /global/scratch file system is not affected by this bug so far. Thus, using /global/scratch instead of /home presents a workaround for this problem. A word of caution though: there is no reason to assume that the same bug could not be triggered on /global/scratch as well. The fact that we have not seen the problem in /global/scratch probably just indicates that the jobs that trigger the bug use the /home filesystem and not /global/scratch.
Brief Optical LOS reported on Western ROADM - Exshaw Node
TICKET INFORMATION:
Subject: Brief Optical LOS reported on Western ROADM - Exshaw Node
Category: Outage
Ticket ID: 20100709-001
Start Time: 2010-07-09 10:54 EDT (2010-07-09 14:54 UTC)
End Time: 2010-07-09 10:55 EDT (2010-07-09 14:55 UTC)
TICKET HISTORY:
== Updated: Malcolm on 2010-07-09 12:31 EDT(2010-07-09 16:31 UTC) ==
Technicians working in the area of the CANARIE fiber were moving fibers and
unintentionally disturbed our fiber, causing the problem.
== Created: Malcolm on 2010-07-09 11:00 EDT(2010-07-09 15:00 UTC) ==
A brief optical loss of signal was reported on on the Western ROADM -
Exshaw node towards Calgary. The failure is took down a number of 10G
circuits, listed below, The cause is unknown and the fibre provider is
being informed.
Affected wavelengths:
VNCN1-CLGR2-01(CANARIE)
VNCN1-CLGR2-02(CANARIE)
STTL1-CLGR2-01(CANARIE)
VNCN1-CLGR2-03(BCnet)
VNCN1-CLGR2-04(WestGrid)
The affected lightpaths are:
10313LP02-GreenStarNet-CLCG2-MTRL2
Neptune300 VCTR-SASK
TRIUMF UBC - McGill
TRIUMF Van-Tor
TRIUMF UBC - UofA
TRIUMF - CERN 5G
RDC VNCV1-CLGR2
NRNet VCTR - SASK
NRNet VCTR - RGNA
NRNet VCTR - EDMN
NRNet VCTR - OTWA
ECONET Van - Mon
ECONET VCTR2-EDMN1
CANARIE NOC
Operations and Engineering
Email: eng@canarie.ca
Weekdays: 08:00-17:00 EDT(UTC-5)
+1.613.944.5612
7/24 pager: +1.613.944.5611
http://www.canarie.ca/canet4/
Brief Optical LOS reported on Western ROADM - Exshaw Node
TICKET INFORMATION:
Subject: Brief Optical LOS reported on Western ROADM - Exshaw Node
Category: Outage
Ticket ID: 20100709-001
Start Time: 2010-07-09 10:54 EDT (2010-07-09 14:54 UTC)
End Time: 2010-07-09 10:55 EDT (2010-07-09 14:55 UTC)
== Created: Malcolm on 2010-07-09 11:00 EDT(2010-07-09 15:00 UTC) ==
A brief optical loss of signal was reported on on the Western ROADM -
Exshaw node towards Calgary. The failure is took down a number of 10G
circuits, listed below, The cause is unknown and the fibre provider is
being informed.
Affected wavelengths:
VNCN1-CLGR2-01(CANARIE)
VNCN1-CLGR2-02(CANARIE)
STTL1-CLGR2-01(CANARIE)
VNCN1-CLGR2-03(BCnet)
VNCN1-CLGR2-04(WestGrid)
The affected lightpaths are:
10313LP02-GreenStarNet-CLCG2-MTRL2
Neptune300 VCTR-SASK
TRIUMF UBC - McGill
TRIUMF Van-Tor
TRIUMF UBC - UofA
TRIUMF - CERN 5G
RDC VNCV1-CLGR2
NRNet VCTR - SASK
NRNet VCTR - RGNA
NRNet VCTR - EDMN
NRNet VCTR - OTWA
ECONET Van - Mon
ECONET VCTR2-EDMN1
CANARIE NOC
Operations and Engineering
Email: eng@canarie.ca
Weekdays: 08:00-17:00 EDT(UTC-5)
+1.613.944.5612
7/24 pager: +1.613.944.5611
http://www.canarie.ca/canet4/
Western ROADM Maintenance at Calgary End
TICKET INFORMATION:
Subject: Western ROADM Maintenance at Calgary End
Category: Scheduled maintenance
Ticket ID: 20100708-007
Start Time: 2010-07-13 22:00 EDT (2010-07-14 02:00 UTC)
End Time: 2010-07-13 23:59 EDT (2010-07-14 03:59 UTC)
== Created: Malcolm on 2010-07-08 15:09 EDT(2010-07-08 19:09 UTC) ==
At the date and time listed, fiber provider will be doing some fiber
cleanup at their colo location in Calgary. They will be cutting over our
dark fiber to a more direct cross connect. This is not expected to last
longer than 30 minutes but the entire window has been reserved.
The following wavelengths will be affected:
VNCN1-CLGR2-01(CANARIE)
VNCN1-CLGR2-02(CANARIE)
STTL1-CLGR2-01(CANARIE)
VNCN1-CLGR2-03(BCnet)
VNCN1-CLGR2-04(WestGrid)
The following lightpaths will be affected:
10313LP02-GreenStarNet-CLCG2-MTRL2
Neptune300 VCTR-SASK
TRIUMF UBC - McGill
TRIUMF Van-Tor
TRIUMF UBC - UofA
TRIUMF - CERN 5G
RDC VNCV1-CLGR2
NRNet VCTR - SASK
NRNet VCTR - RGNA
NRNet VCTR - EDMN
NRNet VCTR - OTWA
ECONET Van - Mon
ECONET VCTR2-EDMN1
CANARIE NOC
Operations and Engineering
Email: eng@canarie.ca
Weekdays: 08:00-17:00 EDT(UTC-5)
+1.613.944.5612
7/24 pager: +1.613.944.5611
http://www.canarie.ca/canet4/
Optical level drop reported on Western ROADM - Exshaw node
TICKET INFORMATION:
Subject: Optical level drop reported on Western ROADM - Exshaw node
Category: Outage
Ticket ID: 20100708-006
Start Time: 2010-07-08 10:20 EDT (2010-07-08 14:20 UTC)
End Time: 2010-07-08 10:21 EDT (2010-07-08 14:21 UTC)
TICKET HISTORY:
== Updated: Malcolm on 2010-07-08 12:14 EDT(2010-07-08 16:14 UTC) ==
Fiber provider has had technician check fibers at site and could see no
problems. There is some building maintenance being done which could have
caused this problem. Ticket will be closed.
== Created: Malcolm on 2010-07-08 10:39 EDT(2010-07-08 14:39 UTC) ==
Optical guage threshold crossing was reported on on the Western
ROADM - Exshaw node. This threshold crossing caused a minor outage on
several of our 10Gig circuits
The cause is unknown and the fibre provider is being
informed.
Affected wavelengths:
VNCN1-CLGR2-01(CANARIE)
VNCN1-CLGR2-02(CANARIE)
STTL1-CLGR2-01(CANARIE)
VNCN1-CLGR2-03(BCnet)
VNCN1-CLGR2-04(WestGrid)
The affected lightpaths are:
10313LP02-GreenStarNet-CLCG2-MTRL2
Neptune300 VCTR-SASK
TRIUMF UBC - McGill
TRIUMF Van-Tor
TRIUMF UBC - UofA
TRIUMF - CERN 5G
RDC VNCV1-CLGR2
NRNet VCTR - SASK
NRNet VCTR - RGNA
NRNet VCTR - EDMN
NRNet VCTR - OTWA
ECONET Van - Mon
ECONET VCTR2-EDMN1
CANARIE NOC
Operations and Engineering
Email: eng@canarie.ca
Weekdays: 08:00-17:00 EDT(UTC-5)
+1.613.944.5612
7/24 pager: +1.613.944.5611
http://www.canarie.ca/canet4/
Optical level drop reported on Western ROADM - Exshaw node
TICKET INFORMATION:
Subject: Optical level drop reported on Western ROADM - Exshaw node
Category: Outage
Ticket ID: 20100708-006
Start Time: 2010-07-08 10:20 EDT (2010-07-08 14:20 UTC)
End Time: 2010-07-08 10:21 EDT (2010-07-08 14:21 UTC)
== Created: Malcolm on 2010-07-08 10:39 EDT(2010-07-08 14:39 UTC) ==
Optical guage threshold crossing was reported on on the Western
ROADM - Exshaw node. This threshold crossing caused a minor outage on
several of our 10Gig circuits
The cause is unknown and the fibre provider is being
informed.
Affected wavelengths:
VNCN1-CLGR2-01(CANARIE)
VNCN1-CLGR2-02(CANARIE)
STTL1-CLGR2-01(CANARIE)
VNCN1-CLGR2-03(BCnet)
VNCN1-CLGR2-04(WestGrid)
The affected lightpaths are:
10313LP02-GreenStarNet-CLCG2-MTRL2
Neptune300 VCTR-SASK
TRIUMF UBC - McGill
TRIUMF Van-Tor
TRIUMF UBC - UofA
TRIUMF - CERN 5G
RDC VNCV1-CLGR2
NRNet VCTR - SASK
NRNet VCTR - RGNA
NRNet VCTR - EDMN
NRNet VCTR - OTWA
ECONET Van - Mon
ECONET VCTR2-EDMN1
CANARIE NOC
Operations and Engineering
Email: eng@canarie.ca
Weekdays: 08:00-17:00 EDT(UTC-5)
+1.613.944.5612
7/24 pager: +1.613.944.5611
http://www.canarie.ca/canet4/
Optical link failure reported on Western ROADM - Exshaw and Calgary Node
TICKET INFORMATION:
Subject: Optical link failure reported on Western ROADM - Exshaw and Calgary Node
Category: Outage
Ticket ID: 20100707-004
Start Time: 2010-07-07 13:34 EDT (2010-07-07 17:34 UTC)
End Time: 0000-00-00 00:00 UTC (0000-00-00 00:00 UTC)
TICKET HISTORY:
== Updated: Malcolm on 2010-07-07 15:31 EDT(2010-07-07 19:31 UTC) ==
Fiber provider has informed us that maintenance crews were moving racks in
the area of our equipment and inadvertently pinched one of our fibers. The
fiber has been replaced and all circuits have lightpaths have restored.
== Updated: Malcolm on 2010-07-07 14:38 EDT(2010-07-07 18:38 UTC) ==
Fiber provider has dispatched technician to the site. Estimated time of arrival
was 1 hour. No ETR could be given.
== Created: Jun on 2010-07-07 13:59 EDT(2010-07-07 17:59 UTC) ==
Optical link failure was reported on on the Western
ROADM - Exshaw and Calgary nodes. The failure is taking
down a number of 10G circuits, listed below,
The cause is unknown and the fibre provider is being
informed.
Affected wavelengths:
VNCN1-CLGR2-01(CANARIE)
VNCN1-CLGR2-02(CANARIE)
STTL1-CLGR2-01(CANARIE)
VNCN1-CLGR2-03(BCnet)
VNCN1-CLGR2-04(WestGrid)
The affected lightpaths are:
10313LP02-GreenStarNet-CLCG2-MTRL2
Neptune300 VCTR-SASK
TRIUMF UBC - McGill
TRIUMF Van-Tor
TRIUMF UBC - UofA
TRIUMF - CERN 5G
RDC VNCV1-CLGR2
NRNet VCTR - SASK
NRNet VCTR - RGNA
NRNet VCTR - EDMN
NRNet VCTR - OTWA
ECONET Van - Mon
ECONET VCTR2-EDMN1
CANARIE NOC
Operations and Engineering
Email: eng@canarie.ca
Weekdays: 08:00-17:00 EDT(UTC-5)
+1.613.944.5612
7/24 pager: +1.613.944.5611
http://www.canarie.ca/canet4/
Silo/Hopper Down Wednesday, July 21, 2010 0900-1300 CST
Silo and Hopper will be down Wednesday, July 21, 2010 from 0900-1300 CST (Saskatoon time) for patching and installation of 10Gb networking in the WestGrid storage infrastructure.
Please contact support@westgrid.ca with any questions or concerns.
