« GoDaddy to ‘rescue’ RegisterFly refugees | Home | Future Plans Regarding BlodgetTech Hosting Services »
Major Outage Affects Our Network.
By Joel Blodgett | November 7, 2007
Some of our clients have asked for information regarding the issues that effected our servers that were down for forty-eight hours over the weekend, with additional incremental problems for another twenty-four hours. In this day and age most companies lease their servers off site from known providers. BlodgetTech is no different. The company we chose was Alabanza, a high end server provided that dealt primarily with hosting providers and other resellers. They provided 24 hour monitoring service and a reliable network. The last outage our company experienced of any type prior to this weekend was an outage around a year ago, and was traced to a physical hardware failure which was repaired and brought back online in less then three hours. Overall, Alabanza had reputable support services, and helped our company grow to where it is today.
Two months ago a company named NaviSite bought Alabanza. Like is common in most mergers NaviSite wanted to reduce cost and put their own systems in place. They decided to migrate our servers from the Alabanza’s Baltimore, MD location to NaviSite’s data center located in Andover, MA. From the start the move was plagued with problems. The proposed dates of the move were pushed on multiple occasions. It was certainly inconvenient to have those time frames pushed it was better to push them versus having it go incredibly bad. On November 2nd at 10PM EDT NaviSite began what was to start a series of blunders that has most likely cost them the Alabanza customer base.
Their original plan was to create virtual copies of all client servers on new equipment in Andover. However, by 5am on the morning of November 3rd they had only accomplished getting a fraction of them transfered. They then proceeded to start physically trucking around 400 of the roughly 900 servers to take a 400 mile trip.
By some time late Saturday night by my accounts, servers started to arrive in NaviSite’s Andover location.
At this point I believe a few of the facts are a blur to me. I do know our server was disconnected from Alabanza’s network at around 5:30pm eastern time on Saturday November 3rd and loaded on a truck. Although I can not be 100% sure as to the exact time I believe it arrived in Andover late Sunday night.
One would think this was the end of the problem but amazingly, it was just the beginning. NaviSite immediately started to power up the servers and connect them to the network. When the final pieces were in place NaviSite switched the network from Baltimore to Andover and plugged it in to the new network. The servers came up for about 10 minutes, and then quickly crashed the network. NaviSite blamed the issue on a DOS (denial of service) attack but later discovered it was “ARP” traffic (Best I understand ARP is excessive communication between the computers inside the network to each other). NaviSite discovered that without the Tipping Point system (think: hardware firewall) they had in Baltimore installed the network would never work. So they proceeded to pack up the Tipping Point box in Baltimore and fly it to Andover. By Sunday night they were installing it but it took several hours and a few up and down of the network to get it to work properly.
Our server finally started to come to life at around 6:30 am on Monday November 5. We spent the better part of the morning repairing a few issues on our managed server to get it back online. By mid morning we had 85% of our customers back online. However, due to routing problems inside the NaviSite network many sites started to have intermittent problems. Our server went back down around 4pm as NaviSite shut down the network for further repairs. Not remembering exact times we were back up a few hours later. By Tuesday morning we were running at or near 100%. We did experience some network issues throughout the day but considering the weekend were fairly stable. We have had no official reports of outages today from our clients.
We will have some new announcements to be made in the near future as we look toward the future and past this very unfortunate incident. We thank you for your patience and understanding during this crisis situation.
Topics: Web Hosting Providers, Alabanza, Server Outage, Navisite, Baltimore, Maryland, Andover, Massachusetts, Boston, Technology, Company News, Company Announcements |
Comments