On May 12, 2015, at 4:50 PM, <tom.p...@csiro.au> <tom.p...@csiro.au> wrote: > > "In practice, for me to be able to replicate and verify your computational > analysis and results, I will need to be able to see your source code, compile > it myself, and potentially modify it." > > Why? In this case to verify something you need to have the same output for a > given input, this does not require any of the above (source code, compilation > or modification), you just require a working version of the program/software.
Just because somebody *says* that their compiled program implements some algorithm does mean it *does*. We have to check, and that requires looking at the source (and in practice, independently compiling it and possibly modifying it). And similarly, just because I have the same input and output as they got, using their compiled program, in no way implies that they actually coded their described algorithm correctly. Again, we have to verify. Now, in principle, if I were a "perfect programmer”, I could just read some C code off the page and know exactly how it will behave for all possible inputs when compiled. But nobody is the perfect programmer, and in practice, to figure out what a piece of code actually does, you have to mess with it, change this and that, fiddle with it. That’s just how real programming works, even for code you have written yourself. In practice, the reality is that to independently verify and replicate your computational results, I must be able to see your code, compile it myself, and modify it. > I like the open source model, but there are problems with this model along > with the others. One major problem is that the authors are expected to > support their software after others have gone in and modified it, which is > unrealistic. I’ve never heard of such a thing. Even if it does exist, there are plenty of open source models that don’t require that. > I just think it is unrealistic to expect the source code just because someone > wrote a paper (again, I like open source, but not everything is destined to > be open source). > > cheers, tom > > Tom Peat > Proteins Group > Biomedical Program, CSIRO > 343 Royal Parade > Parkville, VIC, 3052 > +613 9662 7304 > +614 57 539 419 > tom.p...@csiro.au > > ________________________________________ > From: CCP4 bulletin board [CCP4BB@JISCMAIL.AC.UK] on behalf of Douglas > Theobald [dtheob...@brandeis.edu] > Sent: Wednesday, May 13, 2015 6:06 AM > To: CCP4BB@JISCMAIL.AC.UK > Subject: Re: [ccp4bb] [RANT] Reject Papers describing non-open source software > > On May 12, 2015, at 3:19 PM, Robbie Joosten <robbie_joos...@hotmail.com> > wrote: >> >> I strongly disagree with rejecting paper for any other reasons than >> scientific ones. > > I agree, but … one of the foundations of science is independent replicability > and verifiability. In practice, for me to be able to replicate and verify > your computational analysis and results, I will need to be able to see your > source code, compile it myself, and potentially modify it. These > requirements in effect necessitate some sort of open source model, in the > broadest sense of the term. To take one of your examples, the Ms-RSL license > — I can’t effectively replicate and verify your results if I’m legally > prohibited from compiling and modifying your source code, so the Ms-RSL is > out. > >> A paper describing software should properly describe the >> algorithms to ensure the reproducibility. > > *Should*. In practice, we all know (those programmers among us do, anyway) > that descriptions of source code do not suffice. > >> The source should be available for >> inspection to ensure the program does what was claimed, for all I care this >> can be under the Ms-RSL license or just under good-old copyright. The >> program should preferably be available free for academic users, but if the >> paper is good you should be able to re-implement the tool if it is too >> expensive or doesn't exactly do what you want so it isn't entirely >> necessary. > >> Making the software open source (in an OSS sense) does not solve any >> problems that a good description of the algorithms doesn't do well already. > > This is just wildly wrong. It’s basically impossible to ensure and verify > that a “good" description of the algorithm actually corresponds to the source > code without seeing, using, and modifying the source. To take an > experimental analogy — my lab has endured several cases where we read a > “good" published description of the subcloning and sequencing of some vector, > only to find that the detailed published description is wrong when we are > given the chance to analyze the vector ourselves. It happens all the time, > and computer code is no different in this respect. > >> OSS does not guarantee long-term availability, a paper will like outlive the >> software repository. OSS licenses (not the BSD license) can be so >> restrictive that you end up having to re-implement the algorithms anyway. So >> not having an OSS license should not be a reason to reject the paper about >> the software. >> >> Cheers, >> Robbie >> >>> -----Original Message----- >>> From: CCP4 bulletin board [mailto:CCP4BB@JISCMAIL.AC.UK] On Behalf Of >>> James Stroud >>> Sent: Tuesday, May 12, 2015 20:40 >>> To: CCP4BB@JISCMAIL.AC.UK >>> Subject: Re: [ccp4bb] [RANT] Reject Papers describing non-open source >>> software >>> >>> On May 12, 2015, at 12:29 PM, Roger Rowlett <rrowl...@colgate.edu> >>> wrote: >>> >>>> Was the research publicly funded? If you receive funds from NSF, for >> example, >>> you are expected to share and "make widely available and usable" software >>> and inventions created under a grant (section VI.D.4. of the Award and >>> administration guide). I don't know how enforceable that clause is, >> however. >>> >>> The funding shouldn't matter. I suggest that a publication that has the >> purpose >>> of describing non-open source software should be summarily rejected by >>> referees. In other words, the power is in our hands, not the NSF's.
