I was recently asked some questions about how Kotaku reviews work and figured it's as good a time as ever to provide a refresher.
We use a review system that I explained and implemented in early 2012. It has largely gone unchanged.
It's simple. To a review a game, a Kotaku reviewer writes about a game they played with the mindset of telling a friend about a game and whether it's worth playing. The goal is to keep things casual and say what we feel is true about the game and our feelings about it.
The reviewer can write about the game in any manner that they'd like (really, anything goes) as long as they meet a single requirement: they must include a review box, as seen above. This box provides a capsule take on the game, telling readers whether we'd recommend that they play it (Yes, No, or Not Yet), what we liked and disliked, how much we played it and some other basic info. The box is there in case the reader is in a rush or just wants a short take.
We don't put numerical scores, letter grades or star ratings on our reviews. We don't get included in Metacritic. We've long avoided putting scores on games because, as a team, we're not interested in describing the quality of a piece of artwork in terms of a number (that being said, Van Gogh's Starry Night is clearly a 10/10). The Yes/No/NotYet system is intentionally crude, giving you a casual one/two-word comment about if the game is worth playing, but not standing in for an actual explanation as to why. We believe you need words to know how good a game is.
Some on our team despise review scores. Scores have certainly contributed to some problems for game developers. But I respect the fact that some gamers want to see a number and consider it to be a useful shorthand. It's not one we'll provide, but you can find plenty of numerically-scored game reviews elsewhere.
I want to be clear that the question we're answering with a Yes/No/Not Yet is "Should you play this game?", not "Should you buy this game?" That's because, as I wrote in 2012, we can't presume to know what the value of a dollar is to each of readers. We can only presume to know that you like games enough to want to know if a particular new one is worth playing.
Reviewing a game is not the same as playing a game for fun, but it's on us to ensure that we don't let the pressures of reviewing a game affect our impression of a game. I'm leery of how speeding through a game in order to hit a review deadline might taint the reviewer's take on a game. Good reviewers can account for this, but the risks are always there. Because of that, we don't pledge to always review games by a publisher-selected pre-release embargo date or even by the game's release date. We run most of our reviews by one of those dates, but we're willing to wait if we think that waiting will give us a better sense of a game.
We've been waiting longer to review games that have a significant amount of online gameplay. We'd prefer to play those games on live retail servers against regular gamers. That can't be done until a game is released. To give one recent example of how we handled this, we waited 13 days after the game's release to review Destiny. We waited because the game's servers only went live a day before release. We also waited because our reviewer wanted to judge the game based on real in-the-wild online interaction. We waited longer when we realized that an end-game co-op Raid would be released a week after the game launched, a Raid that turned out to be one of the best parts of the game. The game has continued to evolve since our review ran, but I'm confident that our reviewer had a better understanding of what Destiny had to offer than if he'd run a pre-Raid review on day one.
If a game has flaws that would make us hesitate to recommend it to a friend or reader—or if there are key difference-making modes we haven't been able to play pre-release—we might give it a Not Yet. That's a sign that we think the game might get better through a patch or simply prove to be better the more we play it online. Games with a Not Yet should eventually get a Yes or a No, though, admittedly, we've not been perfect about following up.
Every review indicates how long the reviewer played the game and what modes they tried. We want you to have a sense of how much of the game we've played. Reviews should include screenshots, video and anything else that will help you see what the game actually looks like.
We typically review games using review copies sent to us by developers or publishers. Sometimes we pay for the games ourselves. Publishers that provide us pre-release copies of games will often include some stipulations about when the review can run and what spoilers or other details they'd prefer a review to exclude. Most requests are entirely reasonable (don't give away the ending, for example), but if they're not, we'll push back against the publisher and/or wait until launch day to review the game with zero restrictions. I wrote about the types of requests publishers make here.
We seldom go to review events where publishers have reviewers play the game in, say, rented hotel space. The last one we attended, for Halo 4 in 2012, was local to our NYC-based reviewer. The cons of review events are that you're reviewing a game on the publisher's hardware and under fairly tight deadlines; the pros are that you can try multiplayer with a lot of other people at the event rather than facing empty servers or having to schedule time to play against developers or other reporters online. It's highly unlikely we'd ever travel to a review event; if we did, we'd pay our own way as Kotaku does not accept travel money from publishers.
As a result of our mid-year shift to focusing on post-release coverage, we assign a writer to follow any game we review for at least a month, longer if there's more life to the game. The writer will cover patches, updates, glitches, interesting things happening in the game's community and so on. Usually the person "embedded" in the game will be the reviewer. Not always. And, yes, we'll even follow games that we gave a "No" to, because the game may still be interesting and, hey, other writers on the team might be into the game enough to want to follow it post-release. A game doesn't have to be great for us to care about it; it just has to be interesting.
We review hardware a little differently. For consoles and handhelds, we do actually tell people whether we think they need to have (read: buy) the device. It seems pointless to tell people just to try it, and we imagine that they are significant expenses—and not casual purchases—for most readers. Our reviews of the PS4 and Xbox One still stand at Not Yets. The Wii U is at a Yes. We will re-visit our reviews of the Sony and Microsoft consoles later this month as we did one year after the Wii U's release.
If you have any questions about how we review games, please ask below. You can keep up with our reviews here.
To contact the author of this post, write to stephentotilo@kotaku.com or find him on Twitter @stephentotilo.