Separator

The Problem with Review Scores -- Nine-Point- O: That's Too Low

Default_picture
Friday, October 09, 2009

Editor's note: As the former reviews editor at Electronic Gaming Monthly, I used to spend way too much time thinking about game scores. My blood pressure still shoots up whenever I read the phrase "perfect 10" on a messageboard. -Demian


Numbers are funny things. A number can get someone summarily dismissed, incite people to rage on the Internet (easily), and act as the deciding factor in a purchase. Numbers almost always appear at the end of game reviews, and although those reviews may also contain thousands of descriptive and evaluative words, most gamers seem to focus on the score.

GameSpy Reviews Editor Anthony Gallegos recently wrote a review for Uncharted 2: Among Thieves, and awarded it a 4.5 out of 5. Naturally, a few particularly rage-intoxicated gamers called for him to be raped and killed, in part because score aggregator Metacritic, which converts scores to its own percentage scale to calculate an aggregate score, considered that 4.5 as a 90%. And far too many gamers see anything below a 95% as utter shit.

The key issue here is not the score itself, but the emphasis so many people put on the score. Screw the words and to hell with the context of the article and the written opinion of the author: All that matters is the score.

 

Surely the threats of rape and murder are just hyperbole. But in the real world a threat is a threat. If Starbucks made you a coffee and you thought it was so weak that you storm to the counter and threaten the attendant with sexual assault and death, it's obvious what the real-world outcome would be. You'd need a lawyer.

Gaming sites seem to subtly encourage gamers to affiliate themselves along console lines -- sites such as Giant Bomb, 1UP, Gamespot, and IGN all have console-specific sections and often console-specific forums. On Giant Bomb, you can even choose a specific affiliation that alters the color of your username on their message boards.


A sample comment from GameSpy's Uncharted 2 review.

And it's on forums that most of these battles play out, where gamers will discuss why or how Uncharted 2 could receive a low score of 90, or 9.5, or 4.5, and if that makes it only as good as Resistance 2, not as good as Gears of War 2 with its 100, or 10, or 5.

These numerical values are hardly a science. They're dependent on the author, the website and its scoring system, the time a particular game was released -- and yet they're compared as empirical values. Consider IGN's 1999 Quake 3: Arena review, which rated the graphics at 9.5; obviously, if you were evaluating that game now it would be a different story, unless you were accounting for the the year it was released. Games continue to evolve and develop new ways of interaction, player immersion, storytelling ability, and fidelity, so it's important to understand that scores are in relation to standards at that particular time.

A 10 does not mean that this is the greatest game that ever was, is, or will be. In the same regard, a 9.5 is not bad, nor is a 9 or even an 8.

Console exclusivity is one of the driving factors in online review hysteria -- if you own just one console, you may feel pressure to justify your purchase of both the $60 game and the $300 console to play it on. For some gamers, that console is their team, to defend or cheer on with every news announcement, preview, review, and messageboard thread. It's exhausting just thinking about it.

As soon as Warren Specter's Epic Mickey game was announced as a Nintendo Wii exclusive, hundreds of comment posters claimed the developer, Junction Point Studios, had shot themselves in the foot, how the Wii is shit and on and on, despite Mickey Mouse being a large commercial product in Japan and USA and the Nintendo Wii having an install base of over 50 Million worldwide.

This all makes me very happy to be the type of gamer I am. For years now, I specifically look at a development house, sometimes a specific individual developer, the type of game it is, perhaps read a preview or a few press releases, check out a trailer, and those things inform my purchase decision long before a review even comes out. If I am on the fence and want to read a review, I read the actual words of the article instead of hitting the bottom of the page to see what score it earned.

Unfortunately, gamer rage and fanboy wars seem baked into the Internet as part of gaming culture. And not only do most gamers want to see scores, but a controversial score can mean lots of traffic for website owners. I just take comfort that I am not the only gamer who really does not care about a numerical value between 9 and 10, and the decimal points in-between.

 
Problem? Report this post
BITMOB'S SPONSOR
Adsense-placeholder
Comments (30)
Bitmob_photo
October 08, 2009
My policy is to look at things in a very vague sense. If everything is getting between an 80-100 I can be pretty confident that it's a good game, and if it's getting reviews between a 0 and a 50 consistently I can assume that the game sucks. The gray area(51-79) is where reviews stop being useful. Luckily I listen to numerous podcasts with opinions I respect and use that information to help me along in my decision. All of this only applies to a game from a new developer. My purchasing decisions rely almost entirely on who made it. I will buy anything made by Platinum Games, Team Ico, Naughty Dog, Insomniac, Sucker Punch, and Jupiter. Once they make a game I love, I expect them to continue making games I love. Normally with these studios the reviews confirm what I already know. That's the key thing though, most of the people reading reviews are just reading them to validate their own opinions of the game. If someone is complaining about an Uncharted review, they already think uncharted is awesome, and they don't really need that review to tell them that again.
Jason_wilson
October 08, 2009
I'd be very happy if reviews got rid of scores. I'd support a "buy it/rent it/avoid it" system if people felt it was truly called for, but I'd like people to let the message of the review -- and other coverage -- inform them instead of a score.
Lance_darnell
October 10, 2009
All reviews should also compare the game to a food. So Halo would be a Boston Creme Donut, Ico would be Oatmeal with Syrup, and well, yeah! Great Article. Chris and Jason say it all.
Default_picture
November 16, 2009
I'd like to go without reviews as well. It encourages unfair comparisons between games that have nothing in common. A perfect score doesn't mean it's a perfect game. And like Chris said, reviews with "average" scores are pretty much useless anyway. Why waste my time reading them when I can more in-depth opinions elsewhere? And I still can't believe the reaction to Anthony's review. It was [i]half a star[/i] from being a perfect 5. It blows my mind.
101_1115
November 16, 2009
This is exactly why my site's reviews don't use an alphanumeric score system. Recommended/Research It/Exercise Caution is the scale we use, only because we know some people just can't get through the day without SOME metric attached to it. I totally agree with Jason, otherwise. I would love to see a one-year moratorium on game scores, industry-wide, just as an experiment. I know it will never happen, but it would be a neat exercise regardless, if only to see how many fanboy rage-fueled arguments could be avoided by the absence of that biggest bone of contention.
Jamespic4
November 16, 2009
I love scores. I don't always have the time to read every piece of letter about a game on the Internet, and for some reason every self important writer on the net thinks I ought to. Metacritic just need to teach their team of baboons who to actually interpret a score based on the site's listed grading rubric. Sites without scores should be left off the list altogether. I don't know why it's become so fashionable to naysay scoring among the press. Music, movies, and film have been dealing with it forever.
Franksmall
November 16, 2009
Great piece! I hate how high reviews are now, and how much hyperbole fills almost every review out there. It is like you cannot critique a game, yet still recommend it. Too many high game reviews either just try to ignore any faults of the game in order to justify the high score, or write them off and 'not worth mentioning, but.' Dave Halverson, of Play, is probably the worst at this. His reviews have more hyperbole per paragraph then I ever thought was humanly possible.
101_1115
November 16, 2009
[b]@James:[/b] Last I checked, sites without scores [i]weren't[/i] on Metacritic's list. What would be the point of them including reviews that they can't aggregate into their overall score? In regards to your first point, while I agree that you may not have the time to read every review out there, the reason you [i]should[/i] read more than just the final score is because everyone grades differently. If somebody gave the game lower marks because they felt the story/narrative was trite or poorly executed, but the reader reading the review doesn't give two craps about a game's story, how would they have any idea if they didn't read the whole review? I'd rather give someone's score that I might not agree with the benefit of the doubt by seeing [i]why[/i] they scored it the way they did, rather than just label them a moron because I assume their opinion doesn't jive with mine. Not everybody buys games for the same reasons and, since review scores are subjective to begin with, in the end they really are useless. The only way you'll find out why a person gave it that score is by actually reading the review. To put it a different way (one defending those "self important writers" you mention): imagine you had, say, a report you were working on at your job; something you sunk hours of work into researching and writing. Then imagine that, upon handing it to your boss, he opened to the last page, skimmed the final sentence and declared it "crap", based on that cursory examination alone. Maybe that wouldn't bother you in the least, but it would most people. Mind you, I'm not saying every person writing reviews on the internet (or even in print, for that matter) is amazing at what they do - I just don't agree with being that reductive and painting everyone with the same broad brush.
Bhhdicon_copy
November 16, 2009
Reminds me of the 9.5 that greg Sewart gave Chrono Cross and all the hate he got for that. WTF, right? I'd have given it a 7, all told... Anyways, I'm totally for the Get it/caution/avoid it structure for reviews, however, I'll do a star-rating for every bullet point I cover before the full conclusion is drawn.
Jamespic4
November 16, 2009
@Rob I write game reviews for this site and my own, and I really don't care what anyone has to say about the things I put to paper. You're getting caught up in the idea that in some utopian society, somewhere, people will actually have the time to read everything they come across. They won't; this is prima facie. I completely disagree with your statement about game scores being useless. In fact, I would go so far as to say that this isn't a matter of opinion and that your statement is logically flawed. Clearly, they aren't useless; people make judgments based on them all the time -- for good or ill. I suppose what you mean to say is that people use the grades to distort what is being said in the review. This is certainly true; Gallegos' review of Uncharted (which I hadn't read until I wrote this comment) is totally fair, and the people who criticized obviously didn't take the time to read it. That said, the number he assigns is also appropriate and valid based on his writing. You say that a game with a terrible story being demoted in the end score doesn't matter to someone who doesn't care about the story, but your the recursivity of your logic fails to take into account the sad reality that many people don't read the review. Ultimately, reviewers complaining about how their writing, scores, comments, or whatever may be misperceived comes out of a vanity on the part of the author much in the same way defenses from a game developer ring equally false. It seems to me that you're bemoaning the fact that the person who consumed your entry first, cares about what you wrote, second, understands what you wrote, and third, accepts and agrees with what you wrote. My point is that if they did all of these three things, why did you even bother to write and publish it in the first place? There is nothing you can do to change the fact that the average consumer wants a rating on their product - before, for recommendations, and after, for validation. To me this isn't even a discussion. Things will always be demarcated by their qualities. Everyone does it, even if the way they do it is as intangible as simple thought. To me, Super Mario Sunshine is better than Saw movies...Does that comparison even make sense? Of course not. But there is something ineffably valid about my comparison. It's how I feel. People who review games need to get thicker skin and stop acting like a bunch of wimps. If people don't like and author's review, he or she should take solace in his or her status as an iconoclast. People should be proud to express their ideas, and be willing to defend them to those that they decide don't understand them. In the end, your asking for people to shift the way that they think, act, and consume; it's not going to happen. People will continue to troll, debate grades, and make educated/uneducated decisions or statements on the Internet. Get over it. It's the way it is. P.S. This is the longest comment I've ever written on anything, anywhere. ;)
Default_picture
November 16, 2009
Review scores are inherently dependent not just on the site, but on the reviewer. The important thing is to find reviewers whos likes and dislikes you know and understand, and read their reviews. That will gives you a good basis on which to judge a game. A list of review scores doesn't really mean anything out of context. As for reviewers complaining about a visitor looking at only the score and not reading the review, the solution is simple. STOP GIVING SCORES. Then people will have to read the review to get your opinion. As long as you score things, you feed the score comparing machine.
Picture_002
November 16, 2009
@James I'd say by equal logic, you should get over your annoyance at the "thin skin" of said reviewers. Yours is as self-important an opinion as the other side. You're right, we're not an a utopian society and as such writers have every right to get antsy pants when the lower five percenters do everything with a review but what the author intended. Writers aren't perfect either. That said, James is right in that such is the nature of communication. We lose some degree of ownership of the meaning of a piece the moment we put it out into the world. Funny, I remember presenting the "it's not even a discussion because people are the way they are" argument before to an editor who responded with the question of do we just eliminate discussion on how to make some things better? Really, why don't we shut down universities, stop having world peace talks, and stop trying to develop new medicines because some people are just dumb, people are always going to war, and everyone inevitable dies anyway. That just the way it is. I'm taking James's statement well into hyperbole. I know that. Review scores aren't remotely in the world peace stratosphere. But the discussion is often bigger than the perceived insolvable problem as it does lead to the exchange of ideas and can improve things or at least change [i]some[/i] people. Not all but progress is progress if you feel there's a need for any. Coming down from my marble-white high-horse [insert self-mocking "neighing" here], I've been in discussion not that long before with the staff of a site to which I contribute. I think what tends to happen in the discussion is people tend to contend everything should be catered to them. I don't care for review scores and I really dislike the very existence of what I contend is the very flawed Metacritic, but they have their utility. I had an editor say he thought reviews that went down every feature of a game weren't valuable but while I usually don't read those, I will in case of specific games where it may make or break a decision. For example if the AI in the new entry of a sports series isn't much of an improvement over the version I own, I may not upgrade in spite of better graphics and more bells in whistles in game modes. For the most part, I don't even read reviews much anymore. I feel I glean more from hearing the people I really trust talk about them on podcasts than any review they write. And as I writer I almost prefer to not review games anymore as to just avoid worrying about the mess. I often drop in enough opinions in my columns and editorials. Last time I wrote a comment this long, I made it into an article and somehow it's the most popular thing I've written here. Somehow, I don't think this is as worthy.
Default_picture
November 16, 2009
I'd love to see a total ditching of any scoring system.The walls of text that are written to provide in-depth looks at the game's quality are completely ignored by most people. I would rather have no scoring system, so people can actually READ the review to see the writer's opinion, as well as an opinion justification. I honestly think that most people simply skip the review and look at the number, then rage-quit. At life.
101_1115
November 16, 2009
@James: It's late and I was planning to be in bed about an hour ago. I had a very large reply halfway written out, but I realized that in the end, there was no point to prolonging this. You have your opinion. I have my mine. You have issue with mine and think I'm off-base. I have issue with yours, especially the huge basket of incorrect assumptions you made about me to base your arguments on (who's bringing flawed logic into this again?). And in the end, the world keeps turning. Innit grand? Besides, Gerren said a fair bit of what was in my mouth already anyway, so what's the point in rehashing it. @Gerren: Thanks for saving me some of the hassle, sir.;D
Jamespic4
November 16, 2009
@Rob You're right. But it's fun to discuss!
Default_picture
November 16, 2009
Looking at the above discussion, it appears I'm a little late to the party... In my opinion, I read reviews to get a recommendation. I prefer reviewers that recommend a game to everyone, recommend it to certain groups (RPG fans, series fans, etc), or don't recommend it at all. Just about any reviewer worth their salt has a recommendation line at some point. So far as I can tell, that's about as simple and succinct as a numerical score without being so vague as a 9.0 while still being specific about one's opinion of the game.
Default_picture
November 16, 2009
Wow. Were the Grammar Fairy real I'd be struck down for that last sentence.
Default_picture
November 16, 2009
Heck, I'd love to see scores go the way of the dodo, if for no other reason than making people read and think for themselves. It would certainly cause a trickle effect of improving games as people won't follow the herd because of a score and maybe improve the X-Box Live experience a little because people have to actually think. Okay, the second one's a pipe dream, but I have to try.:P
Fitocrop
November 16, 2009
Just like [b]Mr.McReynolds[/b], I also feel that I'm a bit late to the party, but I would still like to share my thoughts on the discussed topic nevertheless. First and foremost, I would like to make it very clear that I really, really think that [i]people should f$#"ing read more[/i]! Now, I'm [i]not[/i] asking people to read those 5 page reviews from IGN, but it would be nice to see people take interest on at least reading some of the one-page or half-page reviews offered by tons of sites out there in order to put those numbers --or letters-- at the end of them into context. Finding reviewers whose thinking and writting seem to be on the same wavelenght as yours is a wonderful thing. It doesn't mean that they will always share the same opinions with you on every aspect of every game -- or movie, or record, or book -- but it does mean that you've found a source of critical content that you can comeback to because you [i]enjoy[/i] it, and that enriches you by providing you with different points of view on a subject you find interesting. One the dictionary definitions found for the word "Review" on the Merriam-Webster online dictionary states that a Review is: [quote]a critical evaluation (as of a book or play)[/quote] Yes, being an [b]evaluation[/b, a review should assign a value to that which is being evaluated, but it seems that in modern days the vast majority of people can only find value in numbers, for some reason, and completely forget about the value of words. Adjectives such as "eerie", "loose" or "frustrating", [i]say[/i] important things about the quality of the product that's being criticized. If you were to "Wikipedia" the word "Criticism" you'd find the mention of this: [quote]Criticism is the activity of judgement or informed interpretation.[/quote] We cannot interpret the numeric value of something if we are not [i]informed[/i] of the context of which it was extracted from. This said, I find that I agree completely with [b]Mr.Zhang-xing[/b]on this one. [b]@Michael[/b] Great article man. I'd totally give a 9.3 ... just kidding man jaj loved every word of it and of the lenghty discussion that followed it :) Peace out people!
Fitocrop
November 16, 2009
Jeez! I completely messed up my post by manually using the [b]bold[/b] function jajaj I'm sorry guys, all automatic from now on.
Default_picture
November 16, 2009
@Roberto Mr. McReynolds is my dad's name. Please call me T-Mac, T-Mac Attack, or T-Mac attack brought to you by Pep Boys and TIAA Cref. Thanks!
Shoe_headshot_-_square
November 16, 2009
=Back on EGM, I'd get so infuriated at some message board posters (for taking scores out of context...or even text out of context), that some of the guys who worked for me forbade me to ever go on NeoGAF. They knew it'd piss me off too much. :)
Default_picture
November 16, 2009
@Shoe Therein lies the problem. Scores are fine, but people bad. That's why I don't read previews (or previews, or most news...)
Default_picture
November 16, 2009
Reviews I meant before the parentheses...
Jason_wilson
November 16, 2009
@Shoe NeoGAF isn't good for one's health.
Default_picture
November 16, 2009
I think that the written review and the score are both important tools to implement when trying to make a purchasing decision, but I definitely think there is a technique to how to use them. One of the worst things you can do to yourself as video game consumer is to make a purchasing decision based on the score in exclusion to the written review. Doing so only perpetuates the absence of that often-forgotten link between the review, the game, and the gamer: and that's the reviewer. I have always read the written reviews in their entirety so I could try to establish some sort of rapport with the reviewer himself (or herself), and experience the game vicariously through them. Then, if the game sounds interesting enough, I'll get it, and make another pass of the review upon completing the game to see how closely my experience compares to the critic's. After doing this enough times, I'll notice certain reviewer's opinions and reactions to a game are extremely similar to my own. Then, and only then, do I feel confident enough in using that particular reviewer's score alone in making purchasing decisions. This method is akin to establishing a brand loyalty. Only here, the reviewer is the "brand" you put your trust in to deliver a product (his or her opinion) that you will most likely enjoy, despite having conducted minimal research.
Profile_pic4
November 16, 2009
I fear I agree with everyone on certain issues... which if you think about it is actually the point of this post and subsequent comments. I agree that a score is helpful. It gives you a quick and dirty look at something if you have neither time nor patience to read at that moment. Then again, a score is limiting, so I try to delve deeper and do as much reading as I possibly can on most game buying decisions. As an example, I have made several purchases of 6.0 rated games based on a semi-random comment buried in a review about ONE SINGLE BOSS BATTLE, cutscene, music, etc. For me, one single glorious gaming moment of goodness can justify the purchase of a 6.0 game. ESPECIALLY if said game can be had in the bargain basement. As an aside from all the debate, I want to take a moment to applaud Bitmob. There is no other gaming-centric site on the Internets with such consistently insightful and intelligent readers. ... And because I cannot ever allow myself to be this serious for this long.. EVER.. I have to end my comment accordingly.... Poop.
Fitocrop
November 16, 2009
[b]@Keith[/b] I just [i]really[/i] have to back you up on that extra comment about the Bitmob community. I know it's gona sound like I'm putting myself in some sort of high regard --which is not the case at all-- but Bitmob is the only game-related site I have [i]ever [/i]openned a user account on. I'm really glad I did. [b]@Travis[/b] Can I call you [b]Mr.T[/b] ot [b]T-Pain[/b]? ;D
Waahhninja
November 17, 2009
I really like the reviews that allow me to learn about the reviewer's preferences. I know what reviews to take with a grain of salt and which to personally build up. I know what will generally be a day-one purchase (based on developer backgrounds or reliable previews) and which I'll hold off on. I don't know what the Metacritic score was for Onechanbara on 360 but it couldn't have been pretty. I bought it based on D3 and the premise. Reading what someone considers a positive or negative might help clarify some things. For instance, I'm holding off on Brutal Legend until I have the time to sink into it.
Default_picture
November 17, 2009
I'm now starting to think anything to do with scores will generate heated discussion; even without an actual numerical value attached. At least it further reinforced my, until now, faith that I was not the only one that has perhaps... outgrown the need for a score to determine a purchase and it all now based on knowledge of the gaming industry that spurs such consumerism.

You must log in to post a comment. Please register if you do not have an account yet.