So ... stepping back a bit here. Are you saying that even attempting to measure outcomes is harmful, because we might draw the wrong conclusions?
This is a completely new notion to me. You say: "I am reluctant to simply collect some data because I am missing a clear question what we are trying to answer and how the data you want to collect is actually answering that question." I do have a clear question. My question is whether participating in the GSoC contributes to our goal of Community Development. Does it develop our community in any measurable way? I assume, at this point, that it does, or we wouldn't keep doing it. In what way has it had measurable impact on our community? Things that we can clearly measure is adding participants in our project communities, and artifacts added to our projects' revision control repositories. Measuring other things is more difficult, and can be taken as a later task. Easy things first. The larger question of how and what we measure in the Community Development PMC as a whole hinges on this, too. I'm curious if you're resistant to that, too? --Rich On 12/05/2016 04:00 PM, Ulrich Stärk wrote: > On 05.12.16 17:54, Rich Bowen wrote: >> >> >> On 12/05/2016 07:41 AM, Ulrich Stärk wrote: >>>> Or, at the very least, can we make a commitment to track this data going >>>>> forward? >>> Let me play the devil's advocate here: What for? >>> >>> GSoC is completely free for the ASF (on the contrary, we even get a small >>> amount for every accepted >>> student that we can than put towards fulfilling our goals) and as long as >>> we have volunteers willing >>> to organize it and mentor students we can assume that at least those >>> volunteers are seeing value in >>> it. Why the stats other than for satisfying our curiosity? >> >> Or, perhaps, let me give a different answer. >> >> I participated in GSoC as a mentor for $WorkProject. While it didn't >> "cost" me anything in dollars, it cost me probably 200 hours of my time. >> I know that other projects at work put more time in, and some less. >> >> This is an enormous cost to me, as an employee. So it behooves me to >> measure the benefit to the project. A student received payment to write >> code that was discarded. And I (and several of my colleagues) spent a >> huge amount of time, which I could have spent on other things, mentoring >> that student. Benefit to project, pretty heavily negative. > > In that case you should have failed the student pretty early as I tell > mentors every year and as > documented in the mentor guide. > >> >> So, now, here we are at the ASF, doing GSoC with our projects, and >> promoting it to them as a benefit. Does it actually benefit them, or is >> it merely siphoning off time that could be spent on other things. > > If a mentor feels it is the latter, they should immediately fail the student. > >> >> To folks that say we can't measure that, I strongly disagree. There are >> two measures that are obvious and easy. >> >> 1) What % of GSoC student are still active on the project 6 months, 12 >> months, 18 months after the project. (We can debate the definition of >> "active" all you like. > > This implies that the program is only successful if a certain number of > students sticks around. And > this is exactly what I'm arguing against. IMO the program is successful as > soon as a student has > some exposure to one of our communities. > >> >> 2) What % of code developed by GSoC students actually becomes a part of >> the project codebase at the end of the project? > > This has the same problems as trying to measure productivity from code. What > is the percentage > telling us about the question we want to answer? Also see above: If the code > cannot be part of the > project codebase the student should be failed. > >> >> I would maintain that #1 is part of our charter as ComDev, and #2 is >> part of what projects should be made aware of before they sign on. >> >> Again, I'm really not asking for a lot of data here. But I do think that >> it's part of a responsible accounting for participating in GSoC. If, for >> example, it is actually hindering projects, don't we want to know that. >> >> I did *not* participate in GSoC again, on $WorkProject, because it >> clearly hindered my project. >> >> > > I am reluctant to simply collect some data because I am missing a clear > question what we are trying > to answer and how the data you want to collect is actually answering that > question. > > Cheers, > > Uli > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@community.apache.org > For additional commands, e-mail: dev-h...@community.apache.org > -- Rich Bowen - rbo...@rcbowen.com - @rbowen http://apachecon.com/ - @apachecon
signature.asc
Description: OpenPGP digital signature
