Today was certainly one of those days which would live up to an ancient Chinese curse, “May you live in interesting times”. From around 1:20 pm until 5:30 pm we (MSU campus) were separated from the rest of the world. Well, it wasn’t quite that dramatic, but it sure sounds intriguing, right? 🙂 During those 4+ hours we were without Internet connection which has made some operational tasks a bit challenging, but overall we ended up doing all right. Furthermore, being without Internet (for the most part) has made it very clear how dependent we are on it and how important it is in getting certain tasks and initiatives accomplished. Cloud computing can be a blessing and a curse when network fails. But let’s save the talk on the importance of Internet and its benefits/drawbacks for a later date. 😉
As we lost connection to websites located off-campus our networking department began to work diligently on assessing the problem. Considering that we still had full access to all online services located in our data centers we suspected that the issue was an Internet Service Provider (ISP) related problem (in our case, Verizon). It was pretty strange. If you look on all kinds of IT and tech sites, from blogs to o2 Mobile Broadband review, you hardly ever see a whole campus IT network shutting down just like that. It was really inconvenient, so we got to work solving it immediately.
Just to be certain and follow best practices we have inspected our own networking gear, monitoring tools and found them all to be completely functional. Upon this realization, we have escalated the call to our ISP who at first suspected a NJEDge core router as the culprit, but has later informed us that our service interruption was a part of a major Verizon SES (Switched Ethernet Services) outage for all New Jersey business clients. While this is definitely a large problem state wide at least it made us feel a bit better knowing that it’s not just us who are experiencing network difficulties.
Here’s one of the e-mails sent out from our systems and networking group updating users of the status of this issue:
Sent: Thursday, December 04, 2008 3:43 PM
Subject: Update: Internet connectivityTo Our Campus Community-
As a follow up to my previous email, our Internet Service Provider
(Verizon) has indicated that the connectivity problems we are experiencing are the result of a wider, regional outage within Verizon’s New Jersey commercial service area.The Verizon service manager has informed us that the company is performing diagnostic tests on the equipment but did not provide an estimate on when Internet connectivity would be restored.
Verizon seems to be generally reliable and stable, but when they fail, they fail in a big way (similar to Apple products). To all Mac heads out there who just rolled their eyes: I’m sorry, I simply couldn’t resist. 😉
It took them a while to get back online, which is a bit surprising assuming the level or redundancy a company of their size must have, but then again at this time we don’t know the exact event that caused the outage.
Now you know how we spent most of our work day. How was your day, was your business affected by this outage in any way?