TICKET INFORMATION:

Subject: Calgary - Edmonton 10G maintenance
Category: Scheduled maintenance
Ticket ID: 20100706-002
Start Time: 2010-08-25 00:01 MST (2010-08-25 07:01 UTC)
End Time: 2010-08-25 07:00 MST (2010-08-25 14:00 UTC)

== Created: Malcolm on 2010-07-06 13:00 EDT(2010-07-06 17:00 UTC) ==

At the date and times listed a forced fiber relocate is sceduled to take
place. It is not expected to last the entire scheduled window but the entire
window has been reserved.. The Calgary - Edmonton 10G (21197CGED)
will be affected and unavailable for the duration of this window.

The affected lightpaths are:
circuitCore Link Edmonton - Calgary
Core Link Cal-Win
ECONET VCTR2-EDMN1
NRNet VCTR - EDMN
NRNet VCTR - SASK
Neptune300 VCTR-SASK
RDC EDMN1-CLGR2
SRNet backup SASK - RGNA via EDMN
TRIUMF UBC - UofA
WestGrid 10G Cal - Sas
WestGrid Cal - Edm

CANARIE NOC
Operations and Engineering
Email: eng@canarie.ca
Weekdays: 08:00-17:00 EDT(UTC-5)
+1.613.944.5612
7/24 pager: +1.613.944.5611
http://www.canarie.ca/canet4/

TICKET INFORMATION:

Subject: Optical link failure reported on Western ROADM - Exshaw, BC node
Category: Outage
Ticket ID: 20100705-001
Start Time: 2010-07-05 07:15 PDT (2010-07-05 14:15 UTC)
End Time: 2010-07-05 07:16 PDT (2010-07-05 14:16 UTC)

TICKET HISTORY:

== Updated: Thomas on 2010-07-05 16:01 EDT(2010-07-05 20:01 UTC) ==

The provider informed that their technicians were performing clean up
around the PoP in Calgary for the past few days, they might have
accidentally dumped into our fibre. The provider confirmed that the
cleanup was completed. There should have on further interruption.

== Created: Thomas on 2010-07-05 10:41 EDT(2010-07-05 14:41 UTC) ==

Optical link failure was reported on on the Western ROADM - Exshaw,
BC node. The failure took down a number of 10G circuits, listed below,
for a minute. The cause is unknown and the fibre provider is being
informed.

Affected wavelengths:
VNCN1-CLGR2-01(CANARIE)
VNCN1-CLGR2-02(CANARIE)
STTL1-CLGR2-01(CANARIE)
VNCN1-CLGR2-03(BCnet)
VNCN1-CLGR2-04(WestGrid)

The affected lightpaths are:

10313LP02-GreenStarNet-CLCG2-MTRL2
Neptune300 VCTR-SASK
TRIUMF UBC - McGill
TRIUMF Van-Tor
TRIUMF UBC - UofA
TRIUMF - CERN 5G
RDC VNCV1-CLGR2
NRNet VCTR - SASK
NRNet VCTR - RGNA
NRNet VCTR - EDMN
NRNet VCTR - OTWA
ECONET Van - Mon
ECONET VCTR2-EDMN1

CANARIE NOC
Operations and Engineering
Email: eng@canarie.ca
Weekdays: 08:00-17:00 EDT(UTC-5)
+1.613.944.5612
7/24 pager: +1.613.944.5611
http://www.canarie.ca/canet4/

TICKET INFORMATION:

Subject: Optical link failure reported on Western ROADM - Exshaw, BC node
Category: Outage
Ticket ID: 20100705-001
Start Time: 2010-07-05 07:15 PDT (2010-07-05 14:15 UTC)
End Time: 2010-07-05 07:16 PDT (2010-07-05 14:16 UTC)

== Created: Thomas on 2010-07-05 10:41 EDT(2010-07-05 14:41 UTC) ==

Optical link failure was reported on on the Western ROADM - Exshaw,
BC node. The failure took down a number of 10G circuits, listed below,
for a minute. The cause is unknown and the fibre provider is being
informed.

Affected wavelengths:
VNCN1-CLGR2-01(CANARIE)
VNCN1-CLGR2-02(CANARIE)
STTL1-CLGR2-01(CANARIE)
VNCN1-CLGR2-03(BCnet)
VNCN1-CLGR2-04(WestGrid)

The affected lightpaths are:

10313LP02-GreenStarNet-CLCG2-MTRL2
Neptune300 VCTR-SASK
TRIUMF UBC - McGill
TRIUMF Van-Tor
TRIUMF UBC - UofA
TRIUMF - CERN 5G
RDC VNCV1-CLGR2
NRNet VCTR - SASK
NRNet VCTR - RGNA
NRNet VCTR - EDMN
NRNet VCTR - OTWA
ECONET Van - Mon
ECONET VCTR2-EDMN1

CANARIE NOC
Operations and Engineering
Email: eng@canarie.ca
Weekdays: 08:00-17:00 EDT(UTC-5)
+1.613.944.5612
7/24 pager: +1.613.944.5611
http://www.canarie.ca/canet4/

TICKET INFORMATION:

Subject: Calgary-Vancouver multiple 10G circuits flapped
Category: Scheduled maintenance
Ticket ID: 20100702-005
Start Time: 2010-07-02 14:27 PDT (2010-07-02 21:27 UTC)
End Time: 2010-07-02 14:51 PDT (2010-07-02 21:51 UTC)

TICKET HISTORY:

== Updated: Thomas on 2010-07-02 23:02 EDT(2010-07-03 03:02 UTC) ==

It seems to be some kind of fibre disturbance between Glacier and
Golden, BC. The power level was down for 5/6 dB. The provider has been
informed and asked to investigate.

== Created: Thomas on 2010-07-02 21:03 EDT(2010-07-03 01:03 UTC) ==

Multiple Vancouver - Calgary 10G circuits flapped three time within the
time frame shown above. The cause is unknown and being
investigated. The following 10G wavelengths and lightpaths were
affected.

Wavelengths:
WestGrid 10GE Vancouver - Calgary
BCNet 10GE Vancouver - Calgary

lightpaths:
10313LP02-GreenStarNet-CLCG2-MTRL2
Neptune300 VCTR-SASK
TRIUMF UBC - McGill
TRIUMF Van-Tor
TRIUMF UBC - UofA
TRIUMF - CERN 5G
RDC VNCV1-CLGR2
NRNet VCTR - SASK
NRNet VCTR - RGNA
NRNet VCTR - EDMN
NRNet VCTR - OTWA
ECONET Van - Mon
ECONET VCTR2-EDMN1

CANARIE NOC
Operations and Engineering
Email: eng@canarie.ca
Weekdays: 08:00-17:00 EDT(UTC-5)
+1.613.944.5612
7/24 pager: +1.613.944.5611
http://www.canarie.ca/canet4/

TICKET INFORMATION:

Subject: Calgary-Vancouver multiple 10G circuits flapped
Category: Scheduled maintenance
Ticket ID: 20100702-005
Start Time: 2010-07-02 14:27 PDT (2010-07-02 21:27 UTC)
End Time: 2010-07-02 14:51 PDT (2010-07-02 21:51 UTC)

== Created: Thomas on 2010-07-02 21:03 EDT(2010-07-03 01:03 UTC) ==

Multiple Vancouver - Calgary 10G circuits flapped three time within the
time frame shown above. The cause is unknown and being
investigated. The following 10G wavelengths and lightpaths were
affected.

Wavelengths:
WestGrid 10GE Vancouver - Calgary
BCNet 10GE Vancouver - Calgary

lightpaths:
10313LP02-GreenStarNet-CLCG2-MTRL2
Neptune300 VCTR-SASK
TRIUMF UBC - McGill
TRIUMF Van-Tor
TRIUMF UBC - UofA
TRIUMF - CERN 5G
RDC VNCV1-CLGR2
NRNet VCTR - SASK
NRNet VCTR - RGNA
NRNet VCTR - EDMN
NRNet VCTR - OTWA
ECONET Van - Mon
ECONET VCTR2-EDMN1

CANARIE NOC
Operations and Engineering
Email: eng@canarie.ca
Weekdays: 08:00-17:00 EDT(UTC-5)
+1.613.944.5612
7/24 pager: +1.613.944.5611
http://www.canarie.ca/canet4/

Our vendor informed us that our replacement part for the Voltaire switch will not arrive until Friday, July 2. We will therefore reschedule our maintence window for Friday afternoon.  Again, we are sorry for the inconvenience.

Our attempts to revive the IB fabric across orcinus today resulted in more problems, we have lost the connection to the global file systems (home and scratch). We are in contact with HP support, but at the moment, orcinus is not available. The earliest time it will become available for users and job processing is Friday (July 2) afternoon. For the file access hep2.westgrid.ca can be used (scratch is the /global/scratch and home of orcinus is available via /global/home_orcinus)

Breezy and all nodes will be taken offline at 8am, Monday July 5th, to physically relocate the cluster to another room.

The cluster is scheduled to be back online by 12pm, but may be available sooner.

 UPDATE:

The relocation took longer than expected, but the cluster was available again as of 4:30pm. 

UPDATE: We received word that our replacement part will not arrive until tomorrow afternoon (Wed, June 30). At this time, we will notify all users that Orcinus  will be shutting down for maintenance. During this period, there will be no access to the cluster until all required maintenance has been completed.

We hope to restore access to Orcinus as soon as the switch is repaired and we will re-establish full job scheduling once we have verified that the issue has been completely resolved. Again, sorry for this inconvenience.

 Serial jobs are running.
 The parallel jobs which started before Friday, June 25, 6:30 PM (PDT)
 should be running, all other parallel jobs will experience errors and could
 fail, the IB fabric is broken.
 The system wide reservation has been set for Monday, June 28.

 Please observe the behavior of your jobs and let us know if there are any
 problems. Sorry for this inconvenience.

We suffered an issue with our Orcinus IB switch today. Access to the filesystem and some jobs may have been affected. We have restored access; however, our vendor support is looking more deeply into the problem. We hope to restore full functionality as quickly as possible.