It wasn't the numbers that knocked Call of Duty: Elite on its ass the day it went live, it was the questions.
Speaking with Kotaku earlier this week Daniel Suarez, vice president of production at Activision, walked us through how one of the most experienced online gaming companies in the world was blindsided by the demands for their new stat-tracking, community-building Call of Duty service Elite.
A week and a half after the launch of the much ballyhooed service, it remains effectively offline for most gamers, with access prioritized for those paying for the service and some key features turned off.
Initially, Suarez said, it was a myriad of issues that buckled the service.
"We had all sorts of issues: A broken registration system; poor scalability of the service; poor experience trying to login to the web or console and confusion in terms of getting 2XP or your Founders Entitlements," he said. "A lot of this was just core infrastructure issues - mainly tied to specific databases serving up data for the console, web and ultimately mobile (when that's released).
"For lack of a better term - our databases got hammered with the millions of people trying to use Elite and it couldn't put up with the demand."
How could a company so experienced with the massive reach of Call of Duty and with ties to Blizzard, which runs World of Warcraft, underestimate the demands of an online service? Activision consulted with Blizzard, both before the launch and after the problems started, but even their guidance didn't help, Suarez said.
They just did, apparently. Suarez, who called the launch a "pretty rocky start", declined to say just how many people have signed up for either the free or paid versions of Elite.
"Year over year Call of Duty sells tens of millions of units. We had an expectation that a percentage of those users would be interested in Elite (both free and paid)," he said. "We had expected Elite to be popular, but the total amount of people that wanted to be a part of Elite even surprised us—yes, we know, Call of Duty, Blizzard, etc.—we should be prepared. We thought we were."
But the key issue isn't really about the number of people using the service. it's a bit more complicated than that.
The main problem seems to be the database and how information is being pulled off of it. A chief component of Elite is the ability to store detailed player stats and then pull them off of the database remotely in specific ways. These queries are a sort of question that asks the database to sort through the data and send back an answer.
"It's all specific to the data structure within the Elite database and how those queries are called and delivered to the web service, console or mobile," Suarez said. "A call goes into the service to pull something—the database needs to take that data (either game or service or whatever it might be) and twist it, pull it, and deliver it in multiple ways to the end users. Now multiply a complex query by millions—non-stop—for a week. That's the world we live in right now."
Suarez acknowledged that some of the queries weren't created as efficiently as they should have been, but he said the main issue has been that Activision didn't prepare for the pure volume of requests that consoles are delivering.
While Activision ran a beta of Elite, because of the way the service was constructed, they weren't able to properly test the console version. That's because the coding needed to be baked into the game, something that happened with Modern Warfare 3, but not the previous Call of Duty game, Black Ops. So when the beta was live it wasn't really allowing them to test the sort of demand the consoles might have been requiring. That was a fundamental problem.
"Due to the fact that we didn't have a console app on Black Ops, there wasn't a clear picture of user behavior coming from the console," Suarez said. "The beta helped us refine the web experience from a feature set, but really didn't give us all the metrics at scale we needed, specifically the additional load from the console....which was a lot higher than our expectations. This was key contributor to the performance and stability issues consumers have experienced."
Now rolling through the second week of outages and instability, much of Activision's resources have been diverted to working on solutions. And they have been slowing making headway.
Earlier this week a fix rolled out that dealt with the registration bottleneck, now tens of thousands of people are subscribing per an hour, Suarez said.
They company also took the advice of some vocal fans of the service, who are paying for the premium version of Elite.
"We've prioritized premium subscribers into the service via a prioritization queue, so if you are a paid member you get into the console or web automatically," he said. "Free users, due to demand, are put into a queue and allowed in when we have bandwidth or capacity."
On Tuesday Activision was finally able to turn clan creation on. Within 12 hours they had 22,000 clans created.
"As we get more people in we are seeing more features and services stressed, some to their capacity," Suarez said. This means that from time to time over the next week or two, we will be doing maintenance on the service as we bring more people in and stabilize features."
The company plans to roll out the iOS and Android apps for the service before the end of the month.And Call of Duty Elite TV's first episode of Friday Night Fights will air next Friday, Nov. 25.
Wednesday night Suarez was in his office, a line of people out the door, unanswered emails stacking up and a status call looming, as he talked me through the plans to fix the service. It involves daily updates hitting the service, both big and small, and weeks more work.
"It's going to take at least a couple of weeks to get Elite operational to the standards we want," he said. "We are making progress.
"This is not how we wanted to roll out this service, but this is how we need to do it to ensure the best consumer experience for those who can get into the service."