On 05/22/2010 06:41 PM, Daniel Kahn Gillmor wrote: > I'm concerned that the rating metric is wrong in penta when reviewers > leave one of the three categories in a "don't know" state. [...] > Any thoughts on this? Am i misunderstanding something?
OK, i've looked into this further. edrz pointed me toward the SQL that
calculates the scores: sql/views/report/view_report_review.sql
The old wiki page [0] description of the overall score was:
>> The total rating is the average (arithmetic mean) of these three
>> numbers. Of course, this isn't a perfect rating system, but that's why
>> we don't use it directly.
But that is untrue.
I've updated the wiki page [0] to describe the actual calculation:
>> We compute a talk's score in each category by taking the average
>> (arithmetic mean) of all ratings of the talk in that category. A talk's
>> total score is equal to (2*acceptance + actuality + relevance)/4. (so
>> acceptance counts twice as much as the other categories).
>>
>> If no one has rated a talk in a given category (e.g. if everyone has
>> left "actuality" unrated), that category's contribution to the total
>> score is 0.
I actually think this is a reasonable approach, i just didn't understand
what it was doing. So i withdraw my earlier objection.
To be clear, the nice features of this approach are:
* the proportional contributions of the three categories to the overall
score are independent of the number of ratings in each category.
* if a category has no ratings at all, it is as though everyone rated
it zero. If a category has one rating, that is the score used for that
category. So if you don't know, you can let people who do know provide
information without tainting their ratings. And if no one knows, then
there is real ambivalence which is best represented for that category
contributing 0 to the final score.
* i like that acceptance is rated as much as relevance and actuality
put together.
hope this makes sense,
--dkg
[0] http://wiki.debconf.org/wiki/Pentabarf
signature.asc
Description: OpenPGP digital signature
_______________________________________________ Debconf-team mailing list Debconf-team@lists.debconf.org http://lists.debconf.org/mailman/listinfo/debconf-team
