XKCD recently published this humorous comic explaining how you should interpret star ratings online. These are the ratings you come across when browsing online stores (e.g. Amazon) and customer review websites — ratings that supposedly provide an accurate glimpse at how consumers feel about the product. Do they, though?
As the comic shows, the answer is: yes and no.
At one time in its history, YouTube used a 5 star rating system for its videos. In the middle of 2010, however, the service swapped it out in favor of a simple thumbs up/down system. Why? Because the distribution of ratings looked like this:
YouTube product manager Shiva Rajaraman writes,
Seems like when it comes to ratings it’s pretty much all or nothing. Great videos prompt action; anything less prompts indifference. Thus, the ratings system is primarily being used as a seal of approval, not as an editorial indicator of what the community thinks about a video. Rating a video joins favoriting and sharing as a way to tell the world that this is something you love.
Take a look at camera gear reviews on Amazon, and you’ll notice a very similar distribution for most of the products:
Devavrat Shah, a professor of Information and Decision Systems at MIT, says that the flaw of 5 star rating is that they’re too ambiguous and subjective. The MIT News office writes,
One of the problems with basing recommendations on ratings, Shah explains, is that an individual’s rating scale will tend to fluctuate. “If my mood is bad today, I might give four stars, but tomorrow I’d give five stars,” he says. “But if you ask me to compare two movies, most likely I will remain true to that for a while.”
Similarly, ratings scales may vary between people. “Your three stars might be my five stars, or vice versa,” Shah says. “For that reason, I strongly believe that comparison is the right way to capture this.”
Moreover, Shah explains, anyone who walks into a store and selects one product from among the three displayed on a shelf is making an implicit comparison. So in many contexts, comparison data is actually easier to come by than ratings.
Startups are already pouncing on this idea of using comparisons for product recommendations rather than consumer star ratings. Snapsort is one such service. It pits similar cameras head-to-head, and recommends one of them based on the features and specs. User picks are also published, though these aren’t used for the computer-generated recommendations.
It would be interesting to see if any of the major online camera retailers ever ditch the ubiquitous 5-star system with a comparison-based one. It would certainly change the way people shop for photo equipment from the way online shopping is done now to the way people used to buy things in retail stores (i.e. “what’s better between this and that?”).
Until then, go ahead and use the XKCD-recommended guide at the top of this post!