Dear ecologers, The question posed in the quote on fact-checking from David Duffy is: Who should undertake the cost of fact checking? The quote gives 3 answers: warranty by the author, make the data available, in-house by the publisher.
In my experience, as reviewer and in an editorial capacity, one can expect a reviewer to check the facts (results) against interpretation. But it is often difficult for a reviewer to fact check the results, simply because the data are of necessity presented in summary fashion. Here are two examples. If I see a confidence limit that includes zero, where the data are counts, I can be fairly sure as a fact checker that the wrong error distribution was used to compute the ci. If I see a standard error or standard deviation on count data, such that the variance is not at least twice the mean, then I am suspicious, but I can't check it, in the absence of the data. In my view the advent of online data archives are an important step. I would also suggest that journals need to do more than ask the author to warrant that they checked the facts. I would suggest that a journal ask an author how they checked the facts. That's my view. What is your take on who should undertake the cost of fact checking? Best, David S. Quoting Malcolm McCallum <[email protected]>: > It is REALLY easy to screw up a figure, table or number set in a text > if you have no one to review it before submission. IF you are a peer > reviewer, this is one of the things you probably should be looking at. > Do the numbers make sense? > Peer review isn't there just to screen out garbage, its also there to > assist authors. This is especially the case when an editor selects a > reviewer specifically because of their exprtise in a particular area. > I recall once as an editor that I sent a paper that involved some > fancy modeling to a mathematical modeler to review the math. It was > outside of what I did. She said he didn't know anything about the > biology, and I told her that was easily covered by the other two > reviewers, I just wanted to make sure the math was not tom-foolery. > More of this needs to happen in peer review. I see a lot of papers > that misuse different techniques. > > For example, I recall a paper published in one big ecology journal in > which they used baysian statistics, and misinterpreted the sets. They > said something had an effect, when the graph and stats clearly > indicated there was no effect!!! So, the paper ended up widely > covered in the news and people assumed it was what it said, when what > it spent 4-5 pages discussing was complete rubbish. I've also seen > interval analysis used where fuzzy sets should be used, and the misuse > and over-use of monte carlo analysis is just over the top. > > Monte Carlo is only supposed to be used when you have a very great > understanding of the system and very few assumptions and hopefully not > a lot of unpredictable influences. This is actually not all that > common in ecology and environmental work. yet, Monte Carlo is used > and abused by simply "Assuming" things are that might not be. When > you do this with MC you can get VERY wrong answers and there is > virtually no way to check it. Fuzzy approaches are much more rubust > in this regard as is interval analysis. But, you hardly see anyone who > knows how to use these things, or people are caught 20-30 years > out-of-date thinking they are controversial. > > The ideal way to do things is to use fuzzy sets to isolate your data > sets to be used in monte carlo. That way, you reduce the odds of > going completely off tangent. However, no one seems to do this either. > it is pretty amazing because outside of ecology, the alternate > methods are widely applied to many different situtaitons. heck, they > even have fuzzy monte carlo and fuzzy neural networks now. But, that > is an entire different topic. > > The point is, I think it is very reasonable for an editor to select a > peer reviewer form an outside field to check up on methods and > techiques that are outside of his/her expertise, especially if these > are highly technical and particulary novel. A biologist is not always > the best reviewer for some biology papers in such cases. > > On Wed, Apr 30, 2014 at 1:11 PM, David Duffy <[email protected]> wrote: > > "To address this, the publishers of clinical journals must do more to > > ensure that someone takes responsibility for the fact-checking. That could > > involve asking authors to guarantee that they have checked figures, > tables, > > text and abstracts for internal consistency. Publishers could require > > authors to make available suitably anonymized data on each patient as > > metadata to the study, so that readers can trace the source of any > > discrepancy that might creep through. Or the publishers could reach into > > their pockets and provide more in-house resources to perform the necessary > > checking. What is not acceptable is for the situation to continue as it > is, > > with responsibilities undefined and inexact publishing distorting clinical > > messages." > > > > > http://www.nature.com/news/false-positives-1.15119?WT.ec_id=NATURE-20140501 > > > > David Duffy > > > > Pacific Cooperative Studies Unit > > Botany > > University of Hawaii > > 3190 Maile Way > > Honolulu Hawaii 96822 USA > > 1-808-956-8218 > > > > -- > Malcolm L. McCallum, PHD, REP > Department of Environmental Studies > University of Illinois at Springfield > > Managing Editor, > Herpetological Conservation and Biology > > âNothing is more priceless and worthy of preservation than the rich > array of animal life with which our country has been blessed. It is a > many-faceted treasure, of value to scholars, scientists, and nature > lovers alike, and it forms a vital part of the heritage we all share > as Americans.â > -President Richard Nixon upon signing the Endangered Species Act of > 1973 into law. > > "Peer pressure is designed to contain anyone with a sense of drive" - > Allan Nation > > 1880's: "There's lots of good fish in the sea" W.S. Gilbert > 1990's: Many fish stocks depleted due to overfishing, habitat loss, > and pollution. > 2000: Marine reserves, ecosystem restoration, and pollution reduction > MAY help restore populations. > 2022: Soylent Green is People! > > The Seven Blunders of the World (Mohandas Gandhi) > Wealth w/o work > Pleasure w/o conscience > Knowledge w/o character > Commerce w/o morality > Science w/o humanity > Worship w/o sacrifice > Politics w/o principle > > Confidentiality Notice: This e-mail message, including any > attachments, is for the sole use of the intended recipient(s) and may > contain confidential and privileged information. Any unauthorized > review, use, disclosure or distribution is prohibited. If you are not > the intended recipient, please contact the sender by reply e-mail and > destroy all copies of the original message. >
