dear all to collect strengths and weakness of this system of review you can add you comments in the discussion page https://wikimania2016.wikimedia.org/wiki/Talk:Critical_issues_presentations#Feedback_and_Evaluation I have stated reporting your feedback, but please do not hesitate to correct, modify, add.
please consider 1. there are other kind of submissions: go for them! (we are updating them Monday February 8th) https://wikimania2016.wikimedia.org/wiki/Submissions. 2. we are experimenting, surely to make Montreal better than us :) this is a clear objective of Wikimania Esino Lario https://meta.wikimedia.org/wiki/Wikimania_2016_bids/Esino_Lario/Evaluation 3. thank you to all reviewers and people who made submissions!!! iolanda/iopensa > Il giorno 04 feb 2016, alle ore 09:22, WereSpielChequers > <[email protected]> ha scritto: > > Two reviews per submission might work if we had a clear set of criteria that > the reviewers were following and sufficient training of the reviewers that > they were broadly consistent in their marking. But when you get the same > presentation being marked as 5 and 8, as one of mine was then the suspicion > is that the assessors are not working to the same criteria as each other. > That wouldn't matter so much if they were all assessing all submissions, > except that an assessor who varied between 0 and ten points would have far > more influence than assessors who usually voted 6, 7 or 8. But having that > level of inconsistency and only two reviews per submission makes the process > a lottery that depends on who the two reviewers are for your submission. > > As for the content of the reviews, I don't consider that either "5 (average)" > or "6 (rather interesting) tell me anything as to why my submissions were > rejected. > > The other two reviews at least managed one or two lines. One of them even > stretched to two sentences. > > Hope Montreal manages something a bit better, I'm sure either Manilla or > Perth would have done. > > > WereSpielChequers > > > On 3 Feb 2016, at 23:22, Dariusz Jemielniak <[email protected] > <mailto:[email protected]>> wrote: > >> hi, >> >> I have some comments as a person from Academia (and not involved in >> Wikimania process in any way): >> >> 1. Short reviews are definitely not helping in addressing the frustration of >> rejection, yet are quite common in academic peer reviewing, especially for >> conferences. >> >> 2. Double blind peer review (not knowing who is reviewed, and not knowing >> who reviews) is a standard in Academia, although some perceive it as >> contributing to lack of responsibility (especially true in competitive >> journal submissions). >> >> 3. Two reviewers per submission is absolutely on par with the conference >> standards I'm used to. Sometimes there are three, but two is absolutely >> acceptable (although a third opinion should be used if the two disagree too >> much). >> >> 4. It could be useful to sensitize the reviewers that the main purpose of >> the review is to help the author to do better next time. >> >> 5. All this is volunteer work. We should be, generally, grateful to >> reviewers (but in the same time grateful to the contributors, too). >> >> best, >> >> dj >> >> On Wed, Feb 3, 2016 at 5:26 PM, Maarten Dammers <[email protected] >> <mailto:[email protected]>> wrote: >> What kind of ridiculous process is this? This is all I got: >> >> =============== >> >> ----------------------- REVIEW 1 --------------------- >> PAPER: 194 >> TITLE: GLAM+Wikidata >> AUTHORS: Sandra Fauconnier and Maarten Dammers >> >> OVERALL EVALUATION: 8 (Very good) >> >> ----------- REVIEW ----------- >> 8 >> >> >> ----------------------- REVIEW 2 --------------------- >> PAPER: 194 >> TITLE: GLAM+Wikidata >> AUTHORS: Sandra Fauconnier and Maarten Dammers >> >> OVERALL EVALUATION: 6 (Rather interesting) >> >> ----------- REVIEW ----------- >> 6 >> >> ============== >> >> So only two people reviewed this? Who are these people? Why is this secret? >> Last year I had 5 people reviewing my submission [1]. >> >> Maarten >> >> [1] https://wikimania2015.wikimedia.org/wiki/Submission_review/5 >> <https://wikimania2015.wikimedia.org/wiki/Submission_review/5> >> >> Op 3-2-2016 om 23:15 schreef Andy Mabbett: >>> I've just received feedback on one of my pitches saying, in part: >>> >>> "Bad boy Andy! This is supposed to be an anonymous review process, so >>> starting your abstract with your own name, is not entirely fair." >>> >>> -- >>> Andy Mabbett >>> @pigsonthewing >>> http://pigsonthewing.org.uk <http://pigsonthewing.org.uk/> >>> >>> _______________________________________________ >>> Wikimania-l mailing list >>> [email protected] <mailto:[email protected]> >>> https://lists.wikimedia.org/mailman/listinfo/wikimania-l >>> <https://lists.wikimedia.org/mailman/listinfo/wikimania-l> >> >> >> _______________________________________________ >> Wikimania-l mailing list >> [email protected] <mailto:[email protected]> >> https://lists.wikimedia.org/mailman/listinfo/wikimania-l >> <https://lists.wikimedia.org/mailman/listinfo/wikimania-l> >> >> >> >> >> -- >> >> __________________________ >> prof. dr hab. Dariusz Jemielniak >> kierownik katedry Zarządzania Międzynarodowego >> i grupy badawczej NeRDS >> Akademia Leona Koźmińskiego >> http://n <http://www.crow.alk.edu.pl/>wrds.kozminski.edu.pl >> <http://wrds.kozminski.edu.pl/> >> >> członek Akademii Młodych Uczonych Polskiej Akademii Nauk >> członek Komitetu Polityki Naukowej MNiSW >> >> Wyszła pierwsza na świecie etnografia Wikipedii "Common Knowledge? An >> Ethnography of Wikipedia" (2014, Stanford University Press) mojego autorstwa >> http://www.sup.org/book.cgi?id=24010 <http://www.sup.org/book.cgi?id=24010> >> >> Recenzje >> Forbes: http://www.forbes.com/fdc/welcome_mjx.shtml >> <http://www.forbes.com/fdc/welcome_mjx.shtml> >> Pacific Standard: >> http://www.psmag.com/navigation/books-and-culture/killed-wikipedia-93777/ >> <http://www.psmag.com/navigation/books-and-culture/killed-wikipedia-93777/> >> Motherboard: http://motherboard.vice.com/read/an-ethnography-of-wikipedia >> <http://motherboard.vice.com/read/an-ethnography-of-wikipedia> >> The Wikipedian: >> http://thewikipedian.net/2014/10/10/dariusz-jemielniak-common-knowledge >> <http://thewikipedian.net/2014/10/10/dariusz-jemielniak-common-knowledge>_______________________________________________ >> Wikimania-l mailing list >> [email protected] <mailto:[email protected]> >> https://lists.wikimedia.org/mailman/listinfo/wikimania-l >> <https://lists.wikimedia.org/mailman/listinfo/wikimania-l> > _______________________________________________ > Wikimania-l mailing list > [email protected] > https://lists.wikimedia.org/mailman/listinfo/wikimania-l
_______________________________________________ Wikimania-l mailing list [email protected] https://lists.wikimedia.org/mailman/listinfo/wikimania-l
