The survey team spent some time discussing the pros and cons of formats for a number of the questions in the new survey. I think when we initially sent out the first draft of the survey, that specific question was simple checkboxes, as I think it had been in the previous year’s edition. The final survey looks like this:
https://www.evernote.com/l/AL3oM6JyI7JDK4Z33Nk3OplinNhBxgs_G8Y There were several factors that influenced our choices. One is a disinclination to force a strict ranking of items for which ordering isn’t natural. For example, one might favor one resource to discover context for option settings, and another for explanation of error messages — is one more useful than the other? When a respondent had not previously been aware of some of the choices, how does she legitimately rank them compared to others? This overlaps with the realization that stack-ranking of employees is at best contrived and at worst harmful: diverse sets of talents are complementary, and “different” usually != “better”. Another is that the survey is already long, demanding significiant time and energy to thoughtfully complete. Rankings require extra thought and time, which can be in short supply when one just wants to get through the darned thing and get back to work and/or a plate of garlic fries. One contributor observed that in a long survey especially, respondents are likely to just stab at anything to get to the next question, vs. ardously comparing each pair of values (bubble sort!). This phenomenon grows as the number of choices grows. With two choices, it’s not too bad; with 10 it’s a pain. “Okay I’ll rank the first item as 3. Oh but then the fifth item really is 3, so I need to go back and change the first item to, um, 4. But the third item really is 4, so ummm …. " This can lead to lower quality results. Consider Arrow’s theorem: you can’t have an ideally rational democratic social aggregation device. By reducing tedium, we aim to get better results while also improving the experience for the respondent. As we reworked the entire survey, we strove to maximize both value and ease of participation. For certain questions we settled on the matrix format shown at the link above. This lets us capture the value of each resource as an absolute, rather than coercing an (artificial) relative value. It lets the respondent consider the value of each resource independently of the others, while also decreasing tedium. — Anthony > On Jan 27, 2021, at 10:11 PM, Mike Perez <mipe...@redhat.com> wrote: > > Hey Alexandre, > > Sorry for the late reply here. I believe Anthony can give you a response on > why we chose a matrix rating scale type instead of rank. > > — > > — Mike Perez (thingee) > On Wed, Nov 25, 2020 at 8:27 AM Alexandre Marangone <a.marang...@gmail.com> > wrote: > Hi Mike, > > For some of the multiple answer questions like "Which resources do you > check when you need help?" could these be ranked answers instead? It > would allow to see which resources are more useful to the community > > On Tue, Nov 24, 2020 at 10:06 AM Mike Perez <mipe...@redhat.com> wrote: > > > > Hi everyone, > > > > The Ceph User Survey 2020 is being planned by our working group. Please > > review the draft survey pdf, and let's discuss any changes. You may also > > join us in the next meeting *on November 25th at 12pm *PT > > > > https://tracker.ceph.com/projects/ceph/wiki/User_Survey_Working_Group > > > > https://tracker.ceph.com/attachments/download/5260/Ceph%20User%20Survey%202020.pdf > > > > We're aiming to have something ready by mid-December. > > > > -- > > > > Mike Perez > > > > he/him > > > > Ceph / Rook / RDO / Gluster Community Architect > > > > Open-Source Program Office (OSPO) > > > > > > M: +1-951-572-2633 > > > > 494C 5D25 2968 D361 65FB 3829 94BC D781 ADA8 8AEA > > @Thingee <https://twitter.com/thingee> Thingee > > <https://www.linkedin.com/thingee> <https://www.facebook.com/RedHatInc> > > <https://www.redhat.com> > > _______________________________________________ > > ceph-users mailing list -- ceph-users@ceph.io > > To unsubscribe send an email to ceph-users-le...@ceph.io > _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
