October Horror

Posted on October 19, 2007
Filed Under VistaPages Stuff |

The past two weeks have been horrendous. After receiving several complaints about our support team, I set out to hire new level 1, 2 and 3 technicians and had let go of those who were not sharing them same vision of our mission statement. This occured on October 3rd, 2007. 

A few hours after a few of the employee’s were let go, a group of them gained access through means that were unknown to us (which have now been identified and blocked) and inserted malacious code that would corrupt all data within the server if the server were to be rebooted. This code was found and removed from all active servers. Following the removal of the code, we started to experience a large DOS (Read more: denial-of-service) attack. This caused severe disruptions in service for a few days as it was off and on, it came in as spurts. At this point, we had been in talks with our upstream provider and were able to block their initial attempts at flooding our network. They further advanced their attempts and we could no longer fight off the attack with the current network toolset that we had in place — we were forced to make an emergency move of our affected network to a network that would be able to properly mitigate and handle attacks such as the one we were experiencing.

So the move began, Kevin, Ed, and Myself (I had to be there), started the move of the network to our new infrastructure setup, located within MCI/Verizon’s Enterprise datacenter. We had roughly about 6000 web sites down for about 20 minues while we migrated. We deployed several scripts to quickly change the IP’s and then DNS took its sweet time, that caused customers to see their site off and on for the next 24-48 hours. There were several different issues that had to be tweaked as they appeared, which some customers saw downtime. As of Sunday, October 14th, 2007 we were finally past the issues of the attack. It’s a big relief as some of us haven’t slept more than several hours per night to ensure things were getting done quickly, including myself!

I ask that any customer that is still experiencing issues with their web site not loading, please contact technical support at http://support.vistapages.com/, so that we can resolve the issue for you. If you have a ticket already open, simply reply to the ticket and it will be re-opened/updated on our end and an technician will resolve your issue.

What’s been done to prevent this from happenning again? 

Good question! I have been asked that countless number of times. Here’s what’s being done:

Newer, More Sophisticated Network Tools: We now have access to sophisticated network tools that can determine an incoming DOS attack and filter it out to prevent network slowdown. This does not eliminate the threat, it just gives us better information to better fight it off — “A good defence is a good offence”.

Better Backbone/Network: Our shared hosting customers are placed on the world’s #1 IP network by MCI/Verizon. It allows us better uptime, better control, better support. You’ll always get the fastest possible route to us, because of MCI/Verizon’s extensive IP network.

No more remote employee’s: In addition to our regular in-house staff, we use to hire remote staff to handle odd hours, weekends and certain other tasks. No longer will we be employing remote staff. All employee’s will now be working under the same roof under the direct supervision of Steven and Myself.

Longer Telephone Hours / Phone Support: We’ve never had 24/7 telephone support. We’ve started with Mon-Fri 9am to 5pm EST and have extended it to Mon-Fri 9AM to 12AM EST. We will also be extending our telephone hours into the weekend. We are working out a system to provide emergency contact 24/7 in the event of an emergency via telephone, which should be in production soon.

Extensive Audit of Shared & Reseller servers: Our newly appointed team of network and system administrators will be working together to ensure servers are secure, reliable and running efficently. This may mean hardware upgrades to servers and customer migrations. You will be notified before any changes are made to prevent confusion and downtime. This also includes deploying a seperate set of servers to place high usage customers that cause issues for the general population of our customers. We suspend less customers and make other customers happy at the same time. It’s win-win. Some offending users may not be suitable for the high usage servers, and may be required to upgrade to private hosting, vps, dedicated or colocation.

The past two weeks have been a terrible time for us at Vistapages, and we understand that your web site being up and available is the utmost important to you and your business. I want to personally apologize to those that were affected by the issues of the past few weeks and re-assure every customer that changes are on their way to make hosting at Vistapages reliable & safe. For those who have intentionally harmed our company will be and are being dealt with legally and may face criminal prosecution, enough of the evidence is there.

I thank everyone for choosing Vistapages as their web hosting provider.

Sincerely,

Kaumil Patel
President, Vistapages INC.

 

Comments

Leave a Comment

If you would like to make a comment, please fill out the form below.

Name (required)

Email (required)

Website

Comments

1 Comment so far
  1. Ciprian Balus October 25, 2007 2:57 pm

    Good work Kaumil

Recently


Categories


Archives


VistaPages Links