(for tracking purposes, forwarded to pentab...@debconf.org, which goes to the RT instance)
On Thu, May 27, 2010 at 12:40:20PM -0400, Daniel Kahn Gillmor wrote: > On 05/22/2010 06:41 PM, Daniel Kahn Gillmor wrote: > > I'm concerned that the rating metric is wrong in penta when reviewers > > leave one of the three categories in a "don't know" state. > [...] > > Any thoughts on this? Am i misunderstanding something? > > OK, i've looked into this further. edrz pointed me toward the SQL that > calculates the scores: sql/views/report/view_report_review.sql > > The old wiki page [0] description of the overall score was: > > >> The total rating is the average (arithmetic mean) of these three > >> numbers. Of course, this isn't a perfect rating system, but that's why > >> we don't use it directly. > > But that is untrue. > > I've updated the wiki page [0] to describe the actual calculation: > > >> We compute a talk's score in each category by taking the average > >> (arithmetic mean) of all ratings of the talk in that category. A talk's > >> total score is equal to (2*acceptance + actuality + relevance)/4. (so > >> acceptance counts twice as much as the other categories). > >> > >> If no one has rated a talk in a given category (e.g. if everyone has > >> left "actuality" unrated), that category's contribution to the total > >> score is 0. > > I actually think this is a reasonable approach, i just didn't understand > what it was doing. So i withdraw my earlier objection. > > To be clear, the nice features of this approach are: > > * the proportional contributions of the three categories to the overall > score are independent of the number of ratings in each category. > > * if a category has no ratings at all, it is as though everyone rated > it zero. If a category has one rating, that is the score used for that > category. So if you don't know, you can let people who do know provide > information without tainting their ratings. And if no one knows, then > there is real ambivalence which is best represented for that category > contributing 0 to the final score. > > * i like that acceptance is rated as much as relevance and actuality > put together. > > hope this makes sense, > > --dkg > > [0] http://wiki.debconf.org/wiki/Pentabarf > > _______________________________________________ > Debconf-team mailing list > Debconf-team@lists.debconf.org > http://lists.debconf.org/mailman/listinfo/debconf-team _______________________________________________ Debconf-team mailing list Debconf-team@lists.debconf.org http://lists.debconf.org/mailman/listinfo/debconf-team
