Hi Phillippe,
If looking only at the figs. 3A,B,C in the PNAS paper alone, yes, I would
agree with you that the proposed correlation is quite weak. Without the help
of the trend lines, I would probably conclude that there is no correlation
between the IF and number of retractions by a simple look. Of course the R
squares are quite telling on the quality of the fitting already. On the
other hand, the same authors had done similar analysis on a smaller pool of
samples (17 journals) in an earlier study:
http://iai.asm.org/content/79/10/3855.full, figure 1. It seems that when
including way less journals, the trend stood out quite nicely - leading the
authors to say "a strong correlation" in the earlier publication. I am not
sure if the earlier clearer trend was a result of cherry picking, as the
choice of journals looks quite normal – like a standard pool of journals
one particular lab would consider to publish papers on.
It would also be interesting for us on the CCP4BB to try picking only the
journals that we would consider to publish structures on, and plot the RI:IF
graph to see what would happen. Compared to other fields of biology, frauds
in crystallography is probably easier to detect, thus we need worry less
about the false negatives: the low impact papers that were fraudulent or
erroneous, but nobody cared to spend their effort battling.( I think when
taking consideration of this, drawing conclusions from figs. 3A,B,C would
be even more dangerous.)
I would also like to bring two more issues for discussion:
One, in the 2011 IAI paper's fig. 1, the authors plotted Retraction Index,
which is the total # of Retractions multiplied by 1000 then divided by total
number of publication, whereas in the 2012 PNAS paper figures 3A,B,C, the
plots simply used number of retractions to plot against the IFs. I wonder
what they will look like if the figures 3A,B,C were plotted as RI vs IF –
considering that many low or moderately-low IF journals publish huge numbers
of papers.
The second issue is, in the PNAS figures 3A,B,C, at the lower left corner,
although the dots have a dense looking, the viewers have to realize that
most of them only represent 1 to 3 retractions. Ten of such points contain
the same number of retractions that one point at the upper halves of the
panels A and B contains. Maybe simple bar graphs for numbers of retractions
in each IF bin would provide more help. Also, the fact that the averaged IFs
landed at ~8 and ~12 for the fraud and errors cases (fig 3D) suggests that
the absolute number of retractions occurred in high IF journals is quite
significant, especially considering that there are way fewer journals with
IF>10 than the ones with IF<10 in the 200-300 journals. So in my view, maybe
trying to fit a straight line to the distribution is overly idealistic, some
sort of partition does exist.
Zhijie
--------------------------------------------------
From: "DUMAS Philippe (UDS)" <p.du...@ibmc-cnrs.unistra.fr>
Sent: Friday, October 19, 2012 6:15 AM
To: "Zhijie Li" <zhijie...@utoronto.ca>
Subject: Re: [ccp4bb] PNAS on fraud
Le Jeudi 18 Octobre 2012 22:52 CEST, Zhijie Li <zhijie...@utoronto.ca> a
écrit:
Thank you for this funny (and yet significant) comment.
But I do not see clearly whether you agree with me or with the PNAS
paper....
For me, this conclusion in the PNAS paper is just ridiculous.
Philippe D
On curve fitting:
http://twitpic.com/8jd081
--------------------------------------------------
From: "DUMAS Philippe (UDS)" <p.du...@ibmc-cnrs.unistra.fr>
Sent: Thursday, October 18, 2012 1:52 PM
To: <CCP4BB@JISCMAIL.AC.UK>
Subject: Re: [ccp4bb] PNAS on fraud
>
> Le Jeudi 18 Octobre 2012 19:16 CEST, "Bernhard Rupp (Hofkristallrat
> a.D.)"
> <hofkristall...@gmail.com> a écrit:
>
> I had a look to this PNAS paper by Fang et al.
> I am a bit surprised by their interpretation of their Fig. 3: they
> claim
> that here exists a highly signficant correlation between Impact factor
> and
> number of retractations. Personnaly, I would have concluded to a
> complete
> lack of correlation...
> Should I retract this judgment?
> Philippe Dumas
