PC game Heroes of Newerth has suffered a "a catastrophic hardware failure", perhaps the worst we've ever seen for an online title, resulting in the game being completely unplayable. Here's how it happened.
Developers S2 Games have posted a full report on the circumstances leading up to the failure, and it sounds like everything that could have gone wrong, no matter how astronomical the chances, went wrong.
Following the DDoS attacks in May, we retained a world renowned internet security firm to secure our network (this required significant software and architecture changes). After our network was secured, our then current hosting provider began experiencing connectivity issues. On June 9th, we began a transition to one of the premiere hosting companies in the world. Though the transition period was rocky, we had the system in a near 99% functional state. It was at this time that CPU upgrades to the database boxes were needed to allow for additional capacity and consistency of service. On Tuesday, June 21st, the replacement of the CPUs was started. Upon bringing the boxes back online, the engineers noticed anomalies in the raid setup, causing a total loss of data on the master and slave boxes. Following these events, they immediately attempted a restoration process on the data, it is unknown at this time if any of the data is recoverable. Concurrent to this happening, we began work to bring up our previous provider's network that was still in place with the data from June 9th, which brings us to today. We are currently running on a temporary system while we construct our new environment at another hosting firm. Due to this system's temporary nature, not all services may be operating at 100%.
We are planning for another maintenance window at which point we will switch to the new infrastructure. During this transition, we intend on having the game playable and will inform the playerbase of this maintenance at a later time.
Continuing their brutal honesty from earlier in the week, S2 "hold ourselves completely accountable for the events that have transpired", and have promised every player silver and items as compensation.
Considering shit happens, even to the best of us, the mark of a man (or a company!) isn't that mistakes are made, but how people react once they've made one. And S2's transparency has been welcome in an industry that all too often is anything but.