The incident

Up until recently, traffic on TrainBot was light, as it was only open to a handful of clients since it was still in closed beta. Everything was fully operational, and the application as a whole was very responsive. Just a few weeks ago, TrainBot was opened to the public and all is still fine and dandy. To celebrate the launch and promote its online training courses, Baysoft Training Inc. recently offered 75 percent off, on all the training courses. However, this promotional offer caused a sudden influx on TrainBot, far beyond what the company had anticipated. Web traffic shot up by 300 percent and, suddenly, things took a turn for the worse.

Network resources weren't holding up well, server CPUs and memory were at 90-95 percent, and database servers weren't far behind, due to high I/O and contention. As a result, most web requests began to get slower response times, making TrainBot totally unresponsive for most of its first-time clients. It didn't take too long for the servers to crash and for the support lines to get flooded after that.