How Not to Sort by Average Rating (2009)

Averages (even with the post's approach) still have the problem of not being "honest" in the game theory sense. For example, if something is rated 4 stars with 100 reviews, a reviewer who believes its true rating should be 3 stars is motivated to give it 1 star because that will move the average rating closer to his desired outcome. A look at rating distributions shows that this is in fact how many people behave. Median ratings are "honest" in…

While I agree with the author in principle, I think there is an implicit criteria they ignore, which is the intuitive correctness from the perspective of the user. Imagine a user chooses "Sort by rating", and they subsequently observe an item with an average 4.5 ranking above a score of 5.0 because it has a higher Wilson score. Some portion of users will think "Ah, yes, this makes sense because the 4.5 rating is based on many more reviews, therefore…

Why get the lower bound: this means you systematically underestimate the items with fewer ratings. Also this formula assumes normal distribution. There is another solution called 'True Bayesian Average' that is used on IMDB.com, for example. For the formula and the explanation how it works see here: http://answers.google.com/answers/threadview/id/507508.html

Also discussed in Cameron Davidson-Pilon's Bayesian methods for Hackers in the context of Reddit ups/downs: http://nbviewer.ipython.org/github/CamDavidsonPilon/Probabil... For Amazon, though, which is the example in Evan Miller's post, I don't really get why you'd first dichotomize the five-star rating into positive vs. negative and then use Wilson intervals. Just construct a run-of-the-mill 95% confidence interval for the mean of a continuous distribution and sort by the (still plausible) worst case scenario…