On October 1, President Obama took the stage at a press conference and did something unusual for the leader of the free world: He explained why a website wasn't working. Specifically, Obamacare's overloaded online service healthcare.gov, which hadn't had a particularly smooth launch. For a moment, the president sounded indistinguishable from a PR rep for a video game developer like Electronic Arts, Blizzard or Rockstar.
"Like every new product rollout, there are going to be some glitches in the signup process along the way that we will fix. I've been saying this from the start," Obama explained, sounding for all the world like a Blizzard rep, explaining away Diablo III's crippled online launch.
"We're going to be speeding things up in the next few hours to handle all this demand that exceeds anything that we had expected," he continued. Somewhere across the country, former Maxis head Lucy Bradshaw likely nodded in solidarity, recalling when earlier this year she had had offered a very similar explanation in relation to her company's disastrous launch of SimCity.
"It is true that what's happened is the website got overwhelmed by the volume," the President told the Associated Press five days later on Monday the 6th, "and folks are working around the clock and have been systematically reducing the wait times." In an office outside Seattle, ArenaNet president Mike O'Brien may have felt the hairs on the back of his neck stand up, recalling the moment when he halted sales of Guild Wars 2 to keep the game running smoothly.
That same day, top White House technology advisor Todd Park elaborated on the ongoing problems millions of people were having signing in to healthcare.gov. "Right now, we've got what we think we need. The contractors have sent in reinforcements. They are working 24-7. We just wish there was more time in the day." Most gamers have heard that refrain, too.
On October 1, the same day that the Obamacare website went live, Rockstar Games launched GTA Online, the ambitious multiplayer counterpart to their already-ambitious game Grand Theft Auto V. Everyone who'd bought GTA V got GTA Online for free, meaning that the moment Rockstar's servers went live, millions of people attempted to access them and begin playing together at once. As predicted, it was a fiasco. The service was essentially broken; almost no one could log in.
Is anyone going to get this stuff right?
There are a few constants in life: The sun will rise in the morning, and set in the evening. The traffic lane you're in will be moving slower than the one next to you. Movie theater popcorn will always be way too expensive. And if a popular game or online service launches, it will always be broken for at least a week.
At this point, those of us who play video games have come to expect it. If millions of people are excited about an online game, we gird ourselves ahead of time. "I'll have fun playing this… in two weeks, when it actually works," we joke, just a little bit bitterly. And so often, we're proven right. Will that ever not be the case? Or is this simply something that we have to learn to live with? And even if that is the case, shouldn't there be some things we can take for granted? I think there are, and I've listed them here.
First, some background.
Launch-day server overload is old hat for players of massively multiplayer online games like World of Warcraft and Guild Wars. But when Diablo III tripped out of the gate, meeting frustrated players with nothing but endless error messages, it raised our collective awareness of the stability problems that plague large-scale online games.
That's probably because the Diablo series has long been thought of as a single-player game with an online component. Developer Blizzard's decision to give the third game an online requirement was a controversial one. The game's launch was their chance to prove that an always-connected gaming experience was actually good for customers in some unforeseen way.
Suffice to say, it wasn't. Diablo III fell on its face and became the first, best example of why always-online just isn't good for people who play games.
Since then, we've seen it happen plenty more times. Guild Wars 2 had incredibly jammed servers at launch, with ArenaNet's president eventually being true to his word and putting a hold on new sales until they stabilized servers. Final Fantasy XIV couldn't get it right on the second time around, and was a disaster for a good chunk of time after launch. And of course, SimCity was such a colossal, flaming wreck that the SimCity Disaster-Watch tag on our site is still being updated months after the game came out. (In fairness, that tag is being updated mostly with ways the game itself doesn't work; the servers are, for the most part, fine.)
A week after launch, GTA Online appears to be mostly stable, though only yesterday did they manage to get out a patch to address the widespread bug that was deleting players' characters.
However, that does raise the question: Is this the new status quo? Can we expect, from here on out, that every major online launch will be plagued by functionality-killing server overload?
It seems that, at least for now, the answer is a resounding "yes." Rockstar should've been the best candidate to have a smooth launch—they have a huge reservoir of money, and should have theoretically been able to buy enough servers and pay enough technicians to get everything working. Furthermore, since GTA Online launched two weeks after GTA V, they had a pretty good idea of just how many millions of players would attempt to log in on day one. But while they did fix the majority of GTA Online's problems relatively quickly, the fact remains that the game didn't work very well for the better part of a week. If these guys can't get it right, who can?
There is reason for hope: Some online games have managed to pull off relatively failure-free launches. Star Wars: The Old Republic's launch went, as Fahey puts it, "so smoothly that people forget about it." Funcom's two recent games Age of Conan and The Secret World both launched with some minor hitches, but players were able to get in and play from the first day. (That's likely because Funcom learned more than a few hard lessons from the prolonged failure of the Anarchy Online launch.)
The Secret World worked somewhat differently than some other online games, using what Funcom calls a "single server" approach, which puts all players together but separates them out into "dimensions" that they can travel between. While I can't say for sure that that approach would work for every online game—it's entirely possible that other factors contributed to TSW's relatively smooth launch, including the fact that it never really attracted players on the scale of a WoW or a Guild Wars 2—it does appear at least possible to take a different approach to online games that makes it less likely to encounter the sorts of traffic jams that we so often see.
(Side note: I'm not a server expert. For a good breakdown of the gist of how some MMOs function, check out this article from Carbine Studios' lead network engineer David Ray about how the servers work for their still-in-beta MMO Wildstar.)
Those games aren't the only ones to get it right. Activision's massively popular Call of Duty games normally launch to a huge influx of players, and while their launches haven't always gone entirely smoothly—there are usually intermittent crashes and bugs for the first few days of play, though some versions of the game run better than others—their servers don't seem to get hellaciously outmatched as some of the other games in this article. It's a bit apples and oranges—Call of Duty functions very differently online than, say, Guild Wars 2, but still good to see that these games can launch without complete failure.
Activision has launched an annual Call of Duty every year since 2006, so they've probably got this down to something of a science. If other companies had that much practice, would they do a better job, too? And by that reasoning, is there any really excuse for a company to get this wrong more than once?
Valve's Steam service has also come a long way—it wasn't that long ago that it was a complete wreck, marring the 2004 launch of Half-Life 2 with the very sorts of delays and registration woes that today's gamers have become so accustomed to. And yet with practice, Valve too has worked out most of the kinks in their system. Granted, the occasional launch will still encounter some hiccups—I have memories of Portal 2 being harder to play on launch night than it should've been—but they've come a long way since the early days. Practice, it would seem, does make perfect.
The causes behind these failures are as varied as the services and games themselves. Unfortunately but understandably, the moment things go wrong, most video game companies hold technical details close and don't release specifics beyond "We're working hard to increase network stability." When SimCity fell short, its makers got specific enough to say they were increasing both the number and stability of servers.
The nature of most failures, however, is usually evident to anyone who suffers through them. In SimCity's case, the servers were constantly full, and even when they weren't, they were too overworked to provide a stable experience. That's in part because games like SimCity and World of Warcraft all use multiple separate servers, each of which has a population cap. As a result, the individual servers become slammed, and players wind up locked out of various servers, despite having had access to them earlier. As you can imagine, sinking a few hours into a game on a given server, only to return the next day and find that server full, can be pretty annoying.
Back to Obamacare for a second. Given that the Affordable Care Act is a national initiative with, let's be real, a lot more riding on it than some inconvenienced video game fans, there's been more transparency surrounding what went wrong. Tech advisor Park blamed a software problem, not a server or hardware issue—the bulk of the healthcare.gov functions, he said, are ready to handle a huge influx of users, but the software that handles login creation crumbled under the weight of all the new usernames.
That underlines one of the reasons these sorts of failure-at-launch problems can feel so intractable: Massive online programs require a lot of moving parts, and if any one of them fails, the entire system can go down. To hear Park tell it, it's as though healthcare.gov is a well-constructed house with a broken front door. From what Maxis said about SimCity, their front door worked fine, but the house itself didn't have enough rooms. Each house, then, has its own unique structural problems.
In a perfect world, companies would have the online infrastructure to make any online service work as advertised on day one. Clearly, we do not yet live in that perfect world. All the same, there are a few things we should be able to take for granted. Four things, specifically. They're the least we should expect.
1. Let us play offline, at least a little bit.
Sure, GTA Online didn't work all that well. But Rockstar hasn't taken all that much heat for it—certainly not to the extent that Blizzard and EA/Maxis did for Diablo III and SimCity. The reason for that is pretty simple: Grand Theft Auto V is a massive, fun game that players can enjoy offline while waiting for GTA Online to get up and running.
The crucial failing of both Diablo III and SimCity was that they took series that were previously known as offline experiences, made them online-only, and then promptly failed to work when the games launched.
If Diablo III had offered a single-player mode and the optional—though doubtless cool!—multiplayer features hadn't worked for a while after launch, people would have been far less upset. If SimCity had launched with its nifty-sounding online economy and trading functionality broken but a playable offline version working, fewer players would have been demanding refunds, and EA/Maxis would've had to eat far less crow to try to get back into players' good graces. It's a lesson they seem to have finally learned: The latest word is that they're looking into an offline option for SimCity, and they've gone out of their way to confirm that The Sims 4 will be a single-player, offline game.
Massively multiplayer games like Guild Wars 2 and Final Fantasy XIV have a somewhat different row to hoe, given that the entire point of an MMOs is to play online with friends. This next bit may be pie-in-the-sky, but dammit, I'm a dreamer. So: Even those games should theoretically launch with some sort of stripped-down offline option. It could be small: A little area, a character creator, the opportunity to earn some XP and make sure you like your character while waiting to get in and play. It might not be compatible with the way those games currently work—and it could be wide open to exploits and cheaters—but future games could take that kind of thing into account and give players something to do offline.
However they accomplish it, some sort of offline component doesn't seem like too much to ask. It needn't be as grand and complete as the standalone version of GTA V, but players who bought a product deserve to use it, even if just a limited version.
It's heartening to see companies backing off and giving people offline options: To see Diablo III's console version be playable offline, to see SimCity's makers talking about a possible offline version, and an offline Sims 4. And it's particularly nice to see Microsoft do the right thing and relax its stance on the Xbox One's previously announced internet requirement. To this day, I imagine what would've happened if the Xbox One had launched with an internet requirement and hit the same sorts of problems as recent online games. And oh, how I shudder.
2. Do not lose our progress.
Developers, I beseech you: Do not lose our progress. Do not delete our characters. Do not make us use the character creator more than once. Do not allow this to happen. Do not.
Once SimCity got up and running fairly stably, I sank a good five or six hours into it. I built a lovely little city, and was really happy with what I'd accomplished. And then, thanks to some combination of tech and server issues, I was permanently locked out of that city. All those hours, gone.
It should really go without saying, but losing progress due to server failure is unacceptable. Not only does it inconvenience players in an obvious way, it can cause them to lose faith in a game altogether. After losing that first city, I'd begin a new SimCity town only to imagine what would happen if, in a couple of days, I lost it again. My faith in SimCity was fractured beyond repair, and I've never really gotten it back.
Last week, upon getting into GTA Online, I made my first character. Despite the fact that the GTA Online character-creator is almost comically lousy—I can't even tell what my grandparents look like, let alone figure out how many options I have!—I was pretty happy with what I came up with. I named her "Felina." She was a sorta badass biker chick who liked committing crime more than she liked hanging out with friends. I got into the game, watched Felina arrive at Los Santos International Airport, and then the servers failed and I got booted. Upon restarting, I found that Felina hadn't been saved, and I had to start over again.
Other players reported worse failures—losing characters after sinking in dozens of hours and having to start over. (Again, worth noting that yesterday Rockstar said it's finally fixed the problem, though lost characters will stay lost.) If it's too much to demand that all online games work perfectly on day-one—and even if it's too much to ask that they all have some offline component—surely it's not too much to ask that they save our progress locally in some way. As annoying as it is to lose progress online, the worst thing about it (for me, anyway) is how it undermines my faith in the game in general. Why should I invest time in this thing that may just erase all my progress?
3. Level with us. Early.
Even if everything has gone wrong—even if the server farm is on fire, if a family of poltergeists has moved into the engineering offices and a zombie redneck torture family is currently filleting the IT team, a developer or publisher can still come out ahead by following one simple guideline: Tell us what's going on. Level with us. And do it early.
Rockstar's GTA Online launch might have left a lot to be desired in the execution department, but they succeeded in one crucial area: communication. Prior to launch, Rockstar was remarkably forthright about the likelihood of server problems. "There will be the typical growing pains for an online game," they wrote, "including but not limited to crashes, glitches, crazy bugs, gameplay modes and mechanics that need re-balancing and other surprises!" So it was written, and so it came to pass.
In the midst of the rocky launch, Rockstar continued to post an ongoing series of comprehensive, relatively transparent official updates, sharing details on coming fixes and clear timetables for when those fixes would go into effect. If they didn't know precisely when an update would be pushed out, they said they didn't know and gave an estimate.
That kind of clear communication shouldn't be the exception. Compare that to EA/Maxis' weirdly robotic communiqués in the midst of the SimCity fiasco. They referred to SimCity as a success, mentioning fans around the world who were loving the game, and acknowledging the game's catastrophic failures in a smiling "oh, it's not such a big deal" manner that smacked of corporate double-speak. They admitted error in a tone and manner that didn't match up with their customers' feelings of anger and frustration.
"The good news is that SimCity is a solid hit in all major markets," Maxis' Lucy Bradshaw wrote mere sentences after apologizing for the disastrous launch, as if the fact that EA was making money on the game would somehow make us all feel better. "The consensus among critics and players is that this is fundamentally a great game," she continued. Why not just own it? Cut the smiling corporate happy-talk and say, "Hey, we screwed up. We know you're mad, but we're going to make it right. Sorry!" As Rockstar has demonstrated, you don't have to be all gloom and doom about it. You can keep it light. After all, we want the game to work, too. So just be clear and honest with us. We'll thank you for it.
4. Don't make us pay for your services while they're busted.
Players shouldn't be paying for an online service as long as it's not working. If a service is rocky for a week or two, those two weeks should be free, or the game-makers should extend the free trial period to make up for it. If a game is free but allows players to buy in-game items, suspend microtransactions until there's a level playing field (as Rockstar did with GTA Online). In short: If the game is broken, players shouldn't be paying for it.
When it comes to online games and services, consumers are becoming conditioned to expect launch failure. Yet our future will be increasingly connected and online. As our ambitions continue to outpace our technical bandwidth, it's tough not to just throw our hands up in the air and ask, is this stuff ever just going to work?
There's certainly an argument to be made that people only pay attention when things go wrong. Some online games have had relatively smooth launches, and there have been successful technical approaches that indicate that, in fact, some methods work better than others. But given the fact that just last week, Rockstar Games and the Obama White House—two massive organizations with seemingly endless financial and technical resources—both debuted services that were broken at launch, it would appear we still have a ways to go.
Just because things don't always go according to plan doesn't mean it all has to be such a series of fiascoes. If game developers can give us some things we can rely on—something to do offline while we wait for servers to start working, an assurance that we won't lose any progress once we start playing, and clear communication about what's really going on—it'll only help ease our transition to this apparently inevitable brave new online world.