“Process Taser” … Awesome little thing!

Posted on October 28, 2007
Filed Under Random Stuff | Leave a Comment

Over the years, one of the most daunting tasks was to keep the server load to a reasonable level to keep services like apache, mysql, crons running. It’s very difficult to know exactly what people are doing with scripts on their hosting account and when they’ll do it. When an insecure script does get executed it usually shoots the load up to a point where services become locked up — causing slow downs, downtime, etc.

I spoke to our elite bash coder Ed, who then whipped up a little tool which we like to call the “Process Taser”.

Let me explain to you how it works and let’s use the above image to make it easier to explain!

When loads go up, it’s usually because of a customer executing an insecure script, or a script that is not behaving as it should (The student in the blue shirt) causing processes such as Apache to consume the resources of the server; causing the slow down/interruption (the class that was being lectured). The process taser (the fine people with the taser) locates the service that is being hammered, stuns it and restarts the service, causing the load to stay at reasonable load average (often after people are tased, they are in check and behaving as they should).

We have deployed the “Process Taser” on a few of our servers which experience higher loads about two weeks ago as a part of our effort to maintain a stable uptime guarantee for our customers. So far it has worked out to be flawless “knock on wood”. We’re continuing to refine the tool to be better than it already is!

This is not obviously the solution, but this is a tool to enable us to keep the server up while we find, locate and deal with the offending script. Hopefully I did a good job at explaining what I thought was an interesting little tool to increase the performance of our services.

Kaumil Patel
President, Vistapages, Inc.

October Horror

Posted on October 19, 2007
Filed Under VistaPages Stuff | 1 Comment

The past two weeks have been horrendous. After receiving several complaints about our support team, I set out to hire new level 1, 2 and 3 technicians and had let go of those who were not sharing them same vision of our mission statement. This occured on October 3rd, 2007. 

A few hours after a few of the employee’s were let go, a group of them gained access through means that were unknown to us (which have now been identified and blocked) and inserted malacious code that would corrupt all data within the server if the server were to be rebooted. This code was found and removed from all active servers. Following the removal of the code, we started to experience a large DOS (Read more: denial-of-service) attack. This caused severe disruptions in service for a few days as it was off and on, it came in as spurts. At this point, we had been in talks with our upstream provider and were able to block their initial attempts at flooding our network. They further advanced their attempts and we could no longer fight off the attack with the current network toolset that we had in place — we were forced to make an emergency move of our affected network to a network that would be able to properly mitigate and handle attacks such as the one we were experiencing.

So the move began, Kevin, Ed, and Myself (I had to be there), started the move of the network to our new infrastructure setup, located within MCI/Verizon’s Enterprise datacenter. We had roughly about 6000 web sites down for about 20 minues while we migrated. We deployed several scripts to quickly change the IP’s and then DNS took its sweet time, that caused customers to see their site off and on for the next 24-48 hours. There were several different issues that had to be tweaked as they appeared, which some customers saw downtime. As of Sunday, October 14th, 2007 we were finally past the issues of the attack. It’s a big relief as some of us haven’t slept more than several hours per night to ensure things were getting done quickly, including myself!

I ask that any customer that is still experiencing issues with their web site not loading, please contact technical support at http://support.vistapages.com/, so that we can resolve the issue for you. If you have a ticket already open, simply reply to the ticket and it will be re-opened/updated on our end and an technician will resolve your issue.

What’s been done to prevent this from happenning again? 

Good question! I have been asked that countless number of times. Here’s what’s being done:

Newer, More Sophisticated Network Tools: We now have access to sophisticated network tools that can determine an incoming DOS attack and filter it out to prevent network slowdown. This does not eliminate the threat, it just gives us better information to better fight it off — “A good defence is a good offence”.

Better Backbone/Network: Our shared hosting customers are placed on the world’s #1 IP network by MCI/Verizon. It allows us better uptime, better control, better support. You’ll always get the fastest possible route to us, because of MCI/Verizon’s extensive IP network.

No more remote employee’s: In addition to our regular in-house staff, we use to hire remote staff to handle odd hours, weekends and certain other tasks. No longer will we be employing remote staff. All employee’s will now be working under the same roof under the direct supervision of Steven and Myself.

Longer Telephone Hours / Phone Support: We’ve never had 24/7 telephone support. We’ve started with Mon-Fri 9am to 5pm EST and have extended it to Mon-Fri 9AM to 12AM EST. We will also be extending our telephone hours into the weekend. We are working out a system to provide emergency contact 24/7 in the event of an emergency via telephone, which should be in production soon.

Extensive Audit of Shared & Reseller servers: Our newly appointed team of network and system administrators will be working together to ensure servers are secure, reliable and running efficently. This may mean hardware upgrades to servers and customer migrations. You will be notified before any changes are made to prevent confusion and downtime. This also includes deploying a seperate set of servers to place high usage customers that cause issues for the general population of our customers. We suspend less customers and make other customers happy at the same time. It’s win-win. Some offending users may not be suitable for the high usage servers, and may be required to upgrade to private hosting, vps, dedicated or colocation.

The past two weeks have been a terrible time for us at Vistapages, and we understand that your web site being up and available is the utmost important to you and your business. I want to personally apologize to those that were affected by the issues of the past few weeks and re-assure every customer that changes are on their way to make hosting at Vistapages reliable & safe. For those who have intentionally harmed our company will be and are being dealt with legally and may face criminal prosecution, enough of the evidence is there.

I thank everyone for choosing Vistapages as their web hosting provider.

Sincerely,

Kaumil Patel
President, Vistapages INC.

 

Recently


Categories


Archives