LATEST ENTRY

INTERNET | Noah Brier

Rating Systems and Personal Rules

Everyone has different criteria for rating, friending, following ...

February 25, 2009 | RSS | EMAIL | PRINT | 13 COMMENTS

Rating systems are something I find myself discussing fairly frequently. Partly because the idea shows up in a lot of projects and partly because I usually end up relating it to the rules people set for themselves within different social networks.

Let me explain. First off, this is all inspired by something Tyler Cowen wrote recently about rating systems:

The old rating system granted up to five stars but now the maximum number of stars is ten. This signals that they wish to start exaggerating the quality of the product. When there are only five stars you know that they are laying their reputation on the line when they grant five stars to a new CD. (Michelin of course won't give a restaurant more than three stars. They don't calculate out to the fourth decimal place along a scale of one thousand.) If the music isn't good you can decide to stop trusting them. But say they give a new release eight, nine, or who knows maybe eight and a half stars? What exactly are they trying to say? Yes they are putting their reputation on the line when they give ten stars, but this will happen so infrequently that it will be harder to judge their overall trustworthiness.

His point is a good one, but one that is made much easier within a single publication. There is no reason that as part of the change Spin couldn't provide a key to go along with the system, one where they lay out what a 10 really means in terms of past album reviews.

With that said, I don't really want to talk about Spin's rating system, I want to talk about mine. When I rate things within iTunes there are specific rules in my head. If something gets one star, it's pretty damn awesome. That's because only songs that I really like get a star. Therefore, a five star song is my very favorite ever (I don't think I currently have it in rotation, but November Rain might get it). Now I'm sure you have some other way of rating things. Maybe 0 stars is the worst song you've ever heard and you work from there. Whatever it is, the point is, it's probably different than mine. This isn't a big deal when we're the ones consuming our music, after all it's pretty easy to keep track of our own criteria, however, it becomes a lot more confusing when more and more folks are exposed to it.

But exposing folks isn't half the problem, the real trouble starts when we start to build systems that aggregate ratings. At that point, everyone's criteria is thrown into a big bucket, stirred around and then shown off to the world in a way that ultimately means very little. What's a five star video on YouTube? Who the hell knows. For me, the only time I ever rate anything I give it five stars. (For whatever that's worth.)

I'm not sure there's a real point to all this, but I'm going to try and get there. First off, it speaks to implicit rating systems. Knowing the stuff people are actually paying attention to (a la last.fm) is a whole lot more effective than asking people their favorite things. People are generally not that good at telling you what they really like or pay attention to. (I don't have anything to back that up, but it's probably true.)

Also, and this is where things get really tangential, I think this relates to the criteria people have for "friending" on different social networks. Who do you connect with on Facebook? How about LinkedIn? How about Twitter? You probably have thought these things through and answered them for yourself. Equally likely is that my answer is different. We all have different criteria for the decisions we make in these places.

Now none of this really matters and we could all go along happily, except people forget that and get upset when they're not friended, followed or connected with. I get a kick when I read things like this from David Pogue on his adoption of Twitter: "One guy took me to task for asking 'dopey questions.' Others criticized me for various infractions, like not following enough other people, writing too much about nontech topics or sending too many or too few messages." That's insane. (Something Evan Williams acknowledges in the article.) The truth with all this stuff is there are no rules. Or, more accurately, there are an infinite number of them.

So yeah, there's that.


PREVIOUS ENTRY | NEXT ENTRY

LEAVE A COMMENT

First name, first and last, whatever you feel like.

Required, but not displayed (so don't worry about spam).

If you've got one, flaunt it.

You can use some HTML (a's, br's, p's, oh my!) if you'd like, if you don't know what that means, don't worry about it.
REMEMBER ME?

COMMENTS

1barbara

And now you know why we have all those damned standardized tests. Many educators are uncomfortable with the idea of individuals establishing their own criteria for excellence :)

February 26, 2009

2Carito Kanashiro

Probably a more standard rating system is what facebook is doing with the thumbs up. I have this (large) assumption that people only review what is worth reviewing--you either like something very much or just don't give s shit. If you do care, it's because you like it so it's worth giving it the thumbs up. Listo. Done.

So in that sense I tend to trust more the number of views of a youtube clip than the real ratings--it's an indirect measure of how many people did give a shit.

February 26, 2009

3Stephen Landau

What's interesting about iTunes (well, there's plenty that's interesting) is the fact that the rating system is one of the only things you can change when you're listening to tunes on your iPod. So I use the stars more as designations and tagging... one star songs are songs I want to delete from my iPod when I connect it back up to my computer. Two star songs are excluded from my "stuff I haven't heard in 4 weeks" smart playlist (if I'm just not feeling the jam, but want to keep it on my iPod). +3 stars are added into my "+3 star" smart playlist, so that becomes a threshold of "good enough to listen to often." But like you, five star songs are the most awesome.

When rating systems are adjusted to personal usage and technology limitations, their meaning can become irrelevant to "rating" but incredibly important to working around a functional challenge.

Then again, I'm a notoriously stingy grader.

February 26, 2009

4Claire Dalton

"People are generally not that good at telling you what they really like or pay attention to. (I don't have anything to back that up, but it's probably true.)" -- Sure you do. It's just like focus groups. Any number of things can influence what a person says: the other folks in the room, whether or not you think people will agree with you, what they think the questioner wants to hear, etc.

Besides, it's unnatural to be asked for our favorites. Think of the last time you filled out a survey about yourself that asked something like "What CD is playing in your car?" Chances are, it was something like The Mask of Zorro soundtrack but instead you answered Animal Collective because "that's what I'm actually totally all about." Whatever. Moral: I love the honesty of last.fm.

February 26, 2009

5Keith

I think most public rating systems miss out because there is no sense of authority accountability or ownership. The only ratings I really pay attention to are things like Michelin stars, because they stand for something more than some guy in his bathrobe's whim opinion.

As far as the 'friending' stuff goes, I could go on for hours about how the mythical 'social graph' is meaningless ;).

February 26, 2009

6bjorn

You come into his father's kingdom if you are U.S. propagandists
The right to bear arms human right
Just crap in the tax payer
Just slide
Politicians accept the consensus on the term
Left Intellectual = just kill packet
General amnesty

February 27, 2009

7jason

This is interesting. I like the point about facebook's up or down ratings. I've thought for a while that the best aggregate rating systems were boolean. You vote up or down, and maybe are allowed to explain why.

At a glance you can see how many people "like it", "dislike it", or don't care (don't vote) out of total views. It provides a kind of content feedback and addresses the hodgepodge of rating systems people have - or even the various rules one person has depending on mood/day/subject.

February 28, 2009

8Johanna

I agree with Carito; I've noticed that most of the time – unless someone's a "star" rater on Yelp, Amazon, etc. – people tend to rate something if they feel so strongly and sure about it in either direction that they want to either warn or encourage others (a "Trust me, you'll thank me later" type of thing). That is, unless the rating system is made easy/shows up every time (like Netflix "rate this movie"), and eventually ladders up to a benefit to you, i.e. "other movies you'll like, based on your preferences."

I may not be phrasing this next thing 100% right because I really want to think about it more...

One of the reasons I prefer public rating systems over things like Michelin is context. Restaurants don't live in a vacuum: they are part of people's lives. On Yelp, you can look at profiles attached to each review to get a general sense of the context in which the review was written. You can also look at the other reviews that that person has written (and what they wrote them for), and get a general gauge or how the restaurants (or shops) play a part in their everyday lives*, their particular tastes, their "aesthetic" (gross but true), etc. Yakitori Taisho is not on Michelin's NYC Stars 2009, but it would be on mine because of who I am.

* I recently read a review of a Mexican restaurant in Midtown that gave it 1 star. "It was packed and noisy, everyone was drunk /I was there with my boyfriend and wanted to have a romantic time." You can then take the rating with a grain of salt because you have context of that person's night. It wasn't written by a person whose sole job is to review restaurants objectively.

February 28, 2009

9Anjali

Re: rating systems, I remember I took this NYU class once on film criticism and the person teaching the class, a film critic with Tome OutNY, said that many sites/magazines have changed their rating system for films from a 5 star rating to a 6 or 7 or 8 star one, simply because psyhologically, the more stars people see the better they think a film is - even if 5 out of 8 stars means a film isn't as good as 4 out of 5 stars. Which I found rather interesting - I mean we are being manipulated and we don't even know it!!

And yeah, everything is subjective - how we use Facbook, twitter, everything. The sooner people accept that, the better it is for everyone. There is no right way. There's only your way. And that doesn't mean that it's your way or the highway!!

March 4, 2009

10Anjali

I mean TIME Out New York and Facebook, of course. Not 'Tome' and 'Fac'. I'm not illiterate - hahaha!

March 4, 2009

11blaiq

Noah, I disagree with you when you say "the real trouble starts when we start to build systems that aggregate ratings."

I think the exact opposite is true. It is only when these disparate set of personal rules are aggregrated that some real order might actually appear. And it is when these individual ratings are seen in isolation, that they actually mean nothing.

One example James Surowiecki points out in the 'Wisdom of Crowds' is that the average of a crowd's overall guess as to the number of balls in a container (a common promotion game) is usually always better than the closest best estimate by a single person.

Of course, there will be estimates in the guesses that are wildly off the mark on both sides - but, if the aggregation mechanism is right (in this case the simple average works), these extreme scores will cancel themselves out. And what stands out, is amazingly the right answer - or the closest you can get to a right answer.

Of course, the answer only gets better as more and more people join the game and add their guess - or their rating. Regardless of what personal rule or system they are applying.

The problem with public rating systems is the external 'force' that's trying to fool the signal - fans voting multiple times or competition thumbing it down.

I have been meticulously voting movies on IMDb for over ten years now and I follow their aggregated ratings with great interest.

Their IMDb All-time Top 250 is based on votes from regular voters (based on movies voted for and time span as registered user - the exact criteria are secret) - and only movies with at least 1,500 votes from regular voters are eligible.

But even that's not enough to shake off the intenational and unintentional manipulation of the score by crazed fans.

But the best way to find out where a movie stands is to not see where it's at at the time of release but a few years/months later. Even the most die-hard crazy 'socket puppetry wizard' fan eventually moves on to the next craze.

All 3 The Lord of the Rings movies debuted as the top 2-3 movies of all time - very undeservingly of course. So did 'The Dark Knight.'

But, as expected the correction forces have been grindingly at work. The Lord of the Rings movies have been falling steadily and no longer feature in the top 10. The Dark Knight is also headed downwards.

Movies that truly are great show upward momentum even when the craze and money-fuelled publicity has died down. Proof is 'Shawshank Redemption' that has steadlily moved up to the top - and deservingly too. (The entire IMDb list is here - http://www.imdb.com/chart/top)

And all these are based on aggregate ratings of individuals with wildly differing rating systems.

PS: And IMDb also gives you a demographic and score breakdown of the ratings - if you want to seek your own signal in the aggregation. Here's Shawshanks' - http://www.imdb.com/title/tt0111161/ratings

March 4, 2009

12 MV

yeah. i remember watching the video to November Rain and thinking it was an intense, Academy Award winning cinematic masterpiece. Raw emotion, lonesome churches, wet cake.
I was 7.

March 5, 2009

13Catbird

I realize this post is over a week old (i.e. it's practically "elderly" now), but I just saw this:

http://freakytrigger.co.uk/ft/2009/03/which-is-more-broken-music-criticism-or-metacritic/

today and thought it was germane (if I may use a word that I otherwise never use).

March 6, 2009