Why Five Star Ratings Aren’t Very Helpful When Buying Camera Gear Online

XKCD recently published this humorous comic explaining how you should interpret star ratings online. These are the ratings you come across when browsing online stores (e.g. Amazon) and customer review websites — ratings that supposedly provide an accurate glimpse at how consumers feel about the product. Do they, though?

As the comic shows, the answer is: yes and no.

At one time in its history, YouTube used a 5 star rating system for its videos. In the middle of 2010, however, the service swapped it out in favor of a simple thumbs up/down system. Why? Because the distribution of ratings looked like this:

YouTube product manager Shiva Rajaraman writes,

Seems like when it comes to ratings it’s pretty much all or nothing. Great videos prompt action; anything less prompts indifference. Thus, the ratings system is primarily being used as a seal of approval, not as an editorial indicator of what the community thinks about a video. Rating a video joins favoriting and sharing as a way to tell the world that this is something you love.

Take a look at camera gear reviews on Amazon, and you’ll notice a very similar distribution for most of the products:

Devavrat Shah, a professor of Information and Decision Systems at MIT, says that the flaw of 5 star rating is that they’re too ambiguous and subjective. The MIT News office writes,

One of the problems with basing recommendations on ratings, Shah explains, is that an individual’s rating scale will tend to fluctuate. “If my mood is bad today, I might give four stars, but tomorrow I’d give five stars,” he says. “But if you ask me to compare two movies, most likely I will remain true to that for a while.”

Similarly, ratings scales may vary between people. “Your three stars might be my five stars, or vice versa,” Shah says. “For that reason, I strongly believe that comparison is the right way to capture this.”

Moreover, Shah explains, anyone who walks into a store and selects one product from among the three displayed on a shelf is making an implicit comparison. So in many contexts, comparison data is actually easier to come by than ratings.

Startups are already pouncing on this idea of using comparisons for product recommendations rather than consumer star ratings. Snapsort is one such service. It pits similar cameras head-to-head, and recommends one of them based on the features and specs. User picks are also published, though these aren’t used for the computer-generated recommendations.

It would be interesting to see if any of the major online camera retailers ever ditch the ubiquitous 5-star system with a comparison-based one. It would certainly change the way people shop for photo equipment from the way online shopping is done now to the way people used to buy things in retail stores (i.e. “what’s better between this and that?”).

Until then, go ahead and use the XKCD-recommended guide at the top of this post!

  • nombre

    3 stars post

  • Joey Duncan

    I believe it will be flawed no matter what. I restaurant could be called “crap for lunch” leave their food out all day, and buy their ingredients from a gas station quicky mart and people will still give it 4 or 5 stars. People don’t know the difference and as mentioned above aren’t objective enough. I read the average reviews along with the worst ones. If the worst reviews seem like the writes are complete ar$e hats then I ignore them. AND objectively look at the product against the competition. . Make an educated decision and an educated review.

  • Manuel Smirnoff

    I read the evaluations really carefully. There are idiots and there are fanboys. They can all vote from one to five stars. The trick is to read the reviews, and figure out who you want to listen to. Get lazy and only look at the star chart, and you deserve to get screwed.

  • Fra Lippi

    I think a lot of people leave 5 star ratings for the stuff they buy as a way of validating to themselves that they made a good purchasing decision. If they rate it lower then they’re admitting to everyone they made a mistake.

  • Aeropapa

    One thing is for sure.If a writer complains the camera takes “blurry” pictures, I give that reviewer a full moon.

  • hdc77494

    Saw an article yesterday about a company that sells reviews. Authors, or suppliers, buy them in $500 or $1,000 bundles, and they don’t want all five star reviews. It’s about tricking search engines, and making it look like demand is higher than reality.

  • Andy

    Not only that, but I usually will only read the 2,3, and 4 star reviews. As the article mentions, most of the 1 and 5 star reviews tend to have less though put into them. The middle ratings tend to be more thought through.

  • 9inchnail

    Unless the product has the tiniest flaw they don’t like and which other people might not even notice. Then they give 1 star because it’s the worst thing that ever happened to them. I also love people who rate a product with 1 star because they were not satisfied with the seller’s service or stupid stuff like that.

  • Bua

    There is no perfect system. This is the next best.

  • Shover

    You need to read some of reviews to get a better feel of the true rating, if everybody is saying the same thing there might be a problem

  • Jeff Foster

    This is so true. I was reading reviews on the Neewer TT560 flash on Amazon the other day and decided to read through every single review. That flash has enough bad ratings that most people would probably not consider it, but here’s what I found… Two people actually had valid concerns about problems with the item. ALL other low star reviews were posted by people who didn’t know how to operate their camera. So instead of admitting it was their fault, they rated the product poorly.

  • Black Justin Beiber

    Bitter Nikon buyer?

  • 9inchnail

    You all should make a habit of rating reviews. There’s something like “Was this review helpful to you?”. If enough people answer in the negative, stupid reviews might get revised and deleted or at least readers can actually assess if the points made in the review are valid.

  • Trent

    I skip the 5 star ratings and go straight to the 4, 3, 2 star ratings when making an informed decision.

  • Tabby Caat

    A review is written by someone after they have purchased and USED the item. Sales comparison data is created by evaluating the behavior of someone who has not yet used the item and is going solely by the marketing materials – which we all know are often not to be believed. The best data is obtained by READING the reviews. I almost bought some WOOT sale headphones last night, but first I looked for user reviews for that item. It has reviews spread almost equally across the 5 stars, and an average of 3. I clicked thru to read a few of the 1-star ratings and found many people dissatisfied because the item was described as “sweat-proof” rather than “sweat resistant” (like other models) and that it failed when used by those who get sweaty when they exercise, with the controls doing wonky things like repeatedly redialing your last called phone number until you turn your phone off. But the BIG take-away from reading the reviews was that many had also used the prior model and LOVED it, so I looked for the prior model and found it at “clearance prices” for about the same price as the WOOT price. Score! The previous model is no more sweat resistant, but it has much better audio performance. I would have learned NONE of this simply by comparing the packaging and selecting one of 3 choices.

  •!/thelonelylights Adam Cross

    I don’t necessarily look at the rating when looking at new camera equipment – I always read the reviews that people leave along with their rating – their rating and review can often differ as some people will 5 star for a service yet review the product not so highly. I’ve also seen a lot of 1 star ratings for people giving their opinion of a product on Amazon without actually owning the product.