Why Ratings and Reviews Don’t Matter Anymore! (sort of)

This post might ruffle some feathers so I’m gonna be on my best behavior so as not to piss off the jury of my peers.

Here’s my observation: the ratings/reviews on Amazon are absolutely irrelevant. I see you over there eyeing me suspiciously and whittling a piece of driftwood to resemble a pretty spectacular shiv while mumbling, “Tread carefully, Princess Buttercup.”

As a quick aside, yes, I do respond to Buttercup. No, I won’t tell you why.

We hear this all the time: “If you want to help my new book get off the ground, please consider leaving a review on Amazon.” Shit, I do the same thing because to a certain degree (here I’m already going to contradict my earlier thesis, they don’t call me Flip-Flop Buttercup for no reason–but no, I’m still not going to tell you the reason. Stop asking.) reviews lend a degree of social proof. They are that friendly recommendation that says, “This thing right here? Yeah, it’s the shit.” And if that opinion is coming from somebody you trust, Huzzah! You’re likely to pick up the book and give it a shot.

But what if it’s from an untrustworthy source such as… a person on the internet who you’ve never ever even met? Does that opinion matter? Does it carry any weight?

Sure, to an extent. If it’s a well thought out, unbiased critique of the work in question, then absolutely the review can be helpful. If it’s a simple gush-fest of–

“OH MY GOD, I ADORE THIS AND I WANT TO SNEAK INTO THE AUTHOR’S BEDROOM AT NIGHT AND STEAL HIS/HER PILLOW JUST SO I CAN FALL ASLEEP WITH THEIR SWEET MUSK FILLING MY NOSTRILS!”

–Then not so much.

So here’s the problem, I’m cruising around Amazon looking for some sicky gnar gnar new books to check out and give a thorough eye molestation to, and the ratings are so lopsided that they actually impart a negative amount of information.

*This is the part where I’m probably gonna piss off a few people*

In particular, the books with the most lopsided ratings tend to be from self-published authors. What do I mean by this? Well, self-published authors, whether they be fairly popular, or not, tend to have significantly higher ratings than their traditionally published brethren.

Before we get into the why and the how, I want to substantiate this claim with some examples. I spent a little bit of time this morning compiling some datas that I now want to throw in your face. Incoming!

First, I googled top 100 science fiction books of all time. What pops up reads as a who’s who of sci-fi literary mastery. So I just went down the list, took the top 12 titles and searched their Amazon rating to get a baseline. Here we go:

Fahrenheit 451 – 4.2 stars

Ender’s Game – 4.5

Dune – 4.5

1984 – 4.5

Neuromancer – 4.0

Do Androids Dream of Electric Sheep – 4.2

Brave New World – 4.2

War of the Worlds – 4.3

Stranger in a Strange Land – 4.0

Starship Storm Troopers – 4.4

Snow Crash – 4.1

The Left Hand of Darkness – 4.4

Average Rating 4.2

Okay, so pinnacles of the genre have an average rating of 4.2. That’s not so bad, but there is room for improvement.

Next, I went and took the Hugo winners for Best Novel in the past six years. Take a gander below.

Ancillary Justice – 4.2

Equoid – 4.3

Red Shirts – 3.8

Among Others – 3.7

Black Out/All Clear – 3.3

The City & The City – 3.8

The Windup Girl – 3.9

Average Rating – 3.8

Holy 3 star, Batman. This seems low, doesn’t it? Well, no… actually not so much. These books on average score a half-star less than their all-time perennial classic cousins. Which isn’t so terribly far away all things considered.

Now, does it matter that these books are scoring a full star below the holy grail of 5? Nah, not really, because there is an internal consistency among them. Based on these numbers we would expect to see the average “good, perhaps even great” book to rate about 3.2 stars.

But now, let’s introduce some self-publishers into the mix and see how things get all wonkified. Now, I just grabbed some big named self-publishers off the top of my head and went to their author page on Amazon and simply scrolled through the ratings of their most popular books. On average, these books have very solid ratings with over 100 reviewers, but there are certainly some outliers with less. We’ll talk about this later and why it matters.

So, if you’ve been around the self-publishing world for any length of time you’ve probably heard of Johnny B. Truant and Sean Platt (and David Wright who I think might be my all-time favorite curmudgeon). They host a self-publishing podcast that is incredibly popular and jam packed full of actionable advice. If you’re serious about self-publishing you should probably go give them a listen.

Anyways, here are the numbers for their 8 most popular books:

Invasion – 4.4

Contact – 4.6

Robot Proletariat – 4.8

The Beam Season 1 – 4.8

The Beam Season 2 – 4.9

Dream Engine – 4.8

Axis of Aaron – 4.6

Namaste – 4.8

Average Rating: 4.6

Yikes! These guys are consistently scoring a half-star higher than the All-Time Best of the Best Books of the Sci-Fi/Fantasy Genre, and nearly a full star higher than the Hugo awards (which the argument could be made are to be considered some of the best books of their respective years).

Of these books I’ve read The Beam Season One and Robot Proletariat, and you know what, they were both pretty good. They were well written with an engaging story line. I enjoyed them, but you’ll never convince me they are a half-star better than Neuromancer, or Snow Crash, or Do Androids Dream of Electric Sheep.

You can try, but I’m just gonna jam my fingers in my ears and start singing lalalalalala, I can’t hear you. What can I say, I’m a childish debater.

Let’s move onto the next huge Indie name, Dannika Dark. Dannika writes YA and she’s so uber-popular it sickens me. But that’s just the jealousy talking, I’m sure she’s actually very talented.

*Whoosh, ziff, bang!*

Future Anthony here to clean up Past Anthony’s stupid messes! Turns out a wee-bit more research would’ve revealed that Dannika does NOT write YA. In fact, her website explicitly says as much. That’s what I get for skimming! Anyhow, my point still stands. Dannika has a crazy awesome average rating. Thanks to Dannika herself for setting me straight!

*back to the regularly scheduled program*

Let’s take a look at the ratings for her most popular series.

Seven Years – 4.4

Sterling – 4.1

Four Days – 4.9

Three Hours – 4.9

Twist – 4.7

Shine – 4.9

Average Rating: 4.65

Not as good as Johnny and Sean, but still significantly higher than the Hugo Award winners and the All-Time Bestselling SFF. Then again, this is YA, so perhaps it would be more fair to rank it against its genre peers?

For the sake of fairness, let’s compare it against The Hunger Games.

Hunger Games – 4.6

Catching Fire – 4.7

Mocking Jay – 4.3

Average Rating – 4.5

Boom, take that Suzanne Collins you wanna-be hack. Dannika’s got your number–somebody get Ms. Dark a seven figure movie contract, STAT! Then again, that’s only pulling from a sample size of three, which isn’t terribly significant. Hm… okay, let’s see how Dannika stacks up against another little known lady…

J.K. Rowling and the Hairy Clay Potter.

Sorceror’s Stone – 4.8

Chamber of Secrets – 4.7

Goblet of Fire – 4.8

Order of Phoenix – 4.8

Azkabhan – 4.7

Half Blood Prince – 4.4

Deathly Hallows – 4.7

Average Rating – 4.7

Which is pretty much what you would expect for the best selling book about a hairy gardener ever! But, geez, even then it was actually a pretty close match. Rowling only edged out Dark by .05. That’s a slim margin no matter how you slice it.

Is Dark’s series really as good as The Hunger Games? Possibly. Is it almost as good as Harry Potter? Eh.. possibly, but boy I would be surprised.

So, what the hell is going on here? How are self-published authors pulling in such ginormous numbers? Are they cheating? Did they sell their bodies to the Rating Gods in exchange for all those spiky five star reviews?

Maybe? But I think it’s something a little less sinister then that.

There’s a couple things happening here. First, self-published authors recognize the value of good reviews and go waaaay out of their way to field willing reviewers, often reaching out personally to said people. By comparison, when was the last time you got an email from J.K Rowling asking you to review her upcoming book?

Unless you’re secretly a big named newspaper or literary journal simply pretending to be a human in an attempt to infiltrate my blog, then you’ve probably never received jack squat from J.K. And this is important because we have a hard time saying “no” to people we know.

An even harder time saying bad things to, or about, them.

I’ve struggled with the same thing. An author asks me to review their book and because they’ve personally reached out to me, I feel this subconscious pressure to pull my punches and give a slightly better review/rating than I normally would.

I’m gonna go out on a limb here and assume I’m not the only one doing this. So that’s one part of the rating inflation, but there’s a bit more to it than that.

Self-published authors have learned they need to be their own marketing team and they’ve figured out some great ways of doing this starting with email lists/newsletters. This is an awesome way of corralling the people who really love your writing and opening an avenue of communication with them. With the email list an author can send out notifications whenever their next book is dropping. They can send free deals, and short stories, and puppies, and pretty much whatever you can fit into a digital inbox.

This personal connection with the reader goes a long, long ways and we see the effects of it in the startlingly high ratings these authors are pulling in. Let me throw out an example of a book I saw go live the other day and how the author used his preexisting fan base to astounding effect.

Michael Bunker released his book Brother, Frankenstein just a couple days ago. It already has 99 reviews. I think it ended it’s first day with 97 or 98 reviews. How many of those are 5 stars? 88. Fresh out of the gates, less than a week old, and Brother, Frankenstein has nearly a hundred reviews and a 4.9 average.

Whoa. Let that sink in. Michael Bunker used his readership base to do something absolutely impressive.

Now, please don’t interpret me pointing this out as any sort of hate against these other authors (in fact, I picked up a copy of Brother, Frankenstein ’cause I think Bunker is a talented writer and a good storyteller. I’m intrigued to see what all the hype is about!). The indie publishing landscape is brutal. You’ve got to play the cards your dealt and maximize your chances of getting a winner.

Truant, Dark, and Bunker are definitely playing for keeps. Props to them. But the thing is, everybody is doing this and it’s saturating the rating system. It’s hard to find anybody saying a bad thing about Truant or Platt’s work in their reviews. Same with Dark and Bunker. Everything is “THIS IS SO AMAZING I HAVE TO USE CAPS LOCK TO EXPRESS MY ENTHUSIASM!”

That’s not particularly useful to me when I’m searching for a book to read. I’m guessing it’s not very useful for you either, but hey, maybe I’m wrong.

Typically I find the most value in the three star reviews because that seems to be where people are most even-handed in their praise and critique. But shit, some of the books mentioned above don’t even have a single three star, so I’m left in the lurch on that one.

Why’s this a problem?

Oh, I don’t know. I’m probably making something out of nothing. Ultimately we’re all playing for internet points, so does it really matter? Well, yeah, actually. Careers are on the line, after-all.

But here’s the thing: all those five star reviews don’t make me anymore likely to pick up a book. Especially if they are not balanced by a healthy amount of low stars. Look at that All Time SFF list again. Peruse the reviews and you’ll see a substantial portion of readers absolutely hated said books. Does that make them bad? Not at all.

Here’s something we need to realize: A low rating does not mean a book is bad.

Well, to a degree. If it’s significantly below 3 stars, then maybe it is a bad book. But for the most part, a low rating simply means there is something about the book that is polarizing. And that is a good thing. Knowing what people disliked about the book is beneficial and revealing.

I’ll tell you this, I’ve picked up more books (and loved them!) based off a 2 star review trashing said book, than I ever have by a 5 star gush-fest.

What’s the point of all this? Meh, I’ve lost the thread. You tell me. Get down to the comments and tell me what you think!