Dear Engin,
Thank you for your message and for sharing your observations
about pathologies in the processing results produced by various
versions of XDS. This confirms a long-standing concern of ours that we
(i.e. Global Phasing) were perhaps too careful in the way we had so
far limited the dissemination of the results the considerable amount
of work we did on the monitoring of various forms of dysfunction in
releases of XDS from 20240712 onwards. We wanted to avoid being too
publicly provocative towards the developers, so we didn't want to use
the "nuclear option" of raising these issues on the CCP4BB from the
outset without having done our homework first, and also avoid the
discourse being spoiled by knee-jerk or facetious comments, as often
happens with overly "serious" topics. We were also mindful that we
should leave room for the developers themselves to exercise their
prerogative to be the first to publicly forewarn users of the problems
we (and perhaps others) had identified, although in retrospect this
prerogative does not seem to have been exercised.
Instead we used all the connections at our disposal through the
distribution list for autoPROC, which included our Consortium members,
our contacts at those synchrotrons who operate an autoPROC-based
autoprocessing pipeline, and the academic users who have an autoPROC
licence. Only when people reported on the CCP4BB problems that seemed
related to an XDS problem known to us did we discreetly direct them to
our autoPROC Wiki pages documenting our observations and conclusions,
such as the one I included in my recent message. However this clearly
could not reach everyone who uses XDS or relies on processing results
from XDS-based pipelines at synchrotrons, and we were still debating
how to reach these people without generating needless inflammation.
As it happens, I had an opportunity to bring this matter to the
attention of a large gathering of synchrotron software developers at
the ISPyB-MXCuBE Collaboration meeting in Hamburg last week, through a
presentation that Clemens has now attached to his latest Wiki page and
that can be directly accessed at
https://www.globalphasing.com/autoproc/wiki/plugin/attachments/ComparisonProcessing202504/GPhL_ISPyB_2025-1_DESY_GB_Presentation.pdf
Hopefully it can serve as a useful guide, for everyone concerned,
through the contents of all the original Wiki pages (to which links
are provided on Slide 21). It is all essentially work by Clemens for
the autoPROC-based analyses and the documentation of their results,
and by Gleb Bourenkov who used our simulator to generate images with
very low signal that led to the crucial diagnosis of various levels of
bias in integrated intensities (Slides 8, 9 and 11).
Many interesting questions are raised in this factual summary,
that it may now be possible to follow up in discussions on this BB.
With best wishes,
Gerard.
--
On Thu, May 29, 2025 at 12:22:40PM -0500, Engin Özkan wrote:
> To second the observation, we have also seen the extremely sigmoidal
> cumulative intensity distributions from the affected versions of
> XDS, which are accompanied by higher I/s estimations in the higher
> resolution bins, odd-looking Wilson plot, and a bloated resolution
> estimation. In our case, they were not affecting model refinement
> greatly, but we were still worried.
>
> These were observed with mtz files produced by automated beamline
> pipelines, which were also reporting unrealistically high resolution
> estimates, so people who depend on such pipelines (and like the
> higher I/s values) should probably look at their data more
> critically and re-process.
>
> Engin
>
>
> On 5/29/25 12:06 PM, Gerard Bricogne wrote:
> >Dear Phil,
> >
> > I am not sure that I follow you there: perfectly twinned acentric
> >intensities will follow a Chi-squared distribution with four degrees
> >of freedom, while untwinned ones have only two degrees of freedom. The
> >higher the number of degrees of freedom, the more the Chi-squared
> >distribution will peak around its mean value (Central Limit Theorem).
> >This trend is clearly visible in going from the red to the blue curve.
> >
> > The green curve shows an even stronger trend towards a peaked
> >distribution, so actually the data coming our of 20240712 are, so to
> >speak, "hyper-twinned". This shift away from low values is consistent
> >with Gleb Bourenkov's findings, obtained by integrating images with
> >very low signal created with our image simulator, that integrated
> >intensities produced by the six successive buggy versions of XDS since
> >July last year (20240712, 20240723, 20241002, 20250119, 20250224 and
> >20250320) were affected by biases, mostly positive and causing a
> >depletion of the weak intensity population (hence spurious twinning
> >diagnostics), plus an overestimation of I/sig(I) and even Wilson plots
> >curving upwards (!) at high resolution.
> >
> > Please peruse the autoPROC Wiki page for which I sent a link
> >earlier and take a look at the plots it contains: it is all described
> >in there.
> >
> >
> > With best wishes,
> >
> > Gerard.
> >
> >--
> >On Thu, May 29, 2025 at 05:30:54PM +0100, Phil Evans wrote:
> >>Dear Kay
> >>
> >>This is a plot I sent to Eleanor yesterday, from her log file. The
> >>cumulative intensity plot shows I think that there are too few weak
> >>intensities, the opposite of twinning. I don;t know where it comes from but
> >>it’s not right!
> >>Best
> >>Phil
> >>
> >>########################################################################
> >>
> >>To unsubscribe from the CCP4BB list, click the following link:
> >>https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB&A=1
> >>
> >>This message was issued to members of www.jiscmail.ac.uk/CCP4BB, a mailing
> >>list hosted by www.jiscmail.ac.uk, terms & conditions are available at
> >>https://www.jiscmail.ac.uk/policyandsecurity/
> >
> >>
> >>
> >>>On 29 May 2025, at 12:03, Kay Diederichs <kay.diederi...@uni-konstanz.de>
> >>>wrote:
> >>>
> >>>CAUTION: This email originated from outside of the LMB:
> >>>.-owner-ccp...@jiscmail.ac.uk-.
> >>>Do not click links or open attachments unless you recognize the sender and
> >>>know the content is safe.
> >>>If you think this is a phishing email, please forward it to
> >>>phish...@mrc-lmb.cam.ac.uk
> >>>--
> >>>
> >>>Dear Gerard,
> >>>
> >>>part of the problem was probably indeed due to the use of XDS
> >>>BUILT=20240712 (that version was used for CORRECT.LP that Janani Ganesh
> >>>sent to me). I obtained the raw data from Janani Ganesh and processed with
> >>>both the current XDS (BUILT=20250430), and the bad version from July last
> >>>year. My findings:
> >>>- with the bad version the data appear twinned, whereas with the current
> >>>version the data appear untwinned
> >>>- Wilson B is 34 A^2 with the current version, and significantly (and
> >>>unrealistically) lower with the bad version
> >>>
> >>>The structure can be satisfactorily solved and refined with data from the
> >>>current XDS version. With data from the bad version, average B goes down
> >>>to 19 and the R-values stall at high values.
> >>>
> >>>This does not explain the weird B-values that were reported, though. So I
> >>>suspect another problem.
> >>>
> >>>Best,
> >>>Kay
> >>>
> >>>
> >>>
> >>>On Thu, 29 May 2025 11:37:14 +0100, Gerard Bricogne
> >>><g...@globalphasing.com> wrote:
> >>>
> >>>>Dear all,
> >>>>
> >>>> I wrote yesterday a contribution to this thread, but mistakenly
> >>>>used a simple Reply instead of a Group Reply, so that my message did
> >>>>not reach the BB while it intended to, as shown by the introductory
> >>>>"Dear all".
> >>>>
> >>>> I am including it below, as it contains a link to some material
> >>>>that is of general interest but that we had not yet found a way of
> >>>>disseminating to the whole community. Users of XDS, or of XDS-based
> >>>>pipelines, please take notice and do take a close look at this link,
> >>>>and ultimately at the BUILT= information about the exact XDS version
> >>>>used in producing your results - it can make a lot of difference.
> >>>>
> >>>>
> >>>> With best wishes,
> >>>>
> >>>> Gerard.
> >>>>
> >>>>--
> >>>>Date: Wed, 28 May 2025 16:18:25 +0100
> >>>>From: Gerard Bricogne <gb10>
> >>>>Subject: Re: [ccp4bb] Requesting help with twin refinement
> >>>>To: Eleanor Dodson <eleanor.dod...@york.ac.uk>
> >>>>
> >>>>Dear all,
> >>>>
> >>>> These kinds of anomalies (overoptimistic estimates of resolution,
> >>>>unphysical persistence of signal at high resolution, spurious twinning
> >>>>diagnostics) are reminiscent of what we observed with some of the
> >>>>recent versions of XDS, as documented at
> >>>>
> >>>>https://www.globalphasing.com/autoproc/wiki/index.cgi?ComparisonProcessing202504
> >>>>
> >>>>and three predecessor pages referenced from within this one.
> >>>>
> >>>> Only one version is currently available from the XDS download
> >>>>page (that we consider as being OK) but there was a succession of many
> >>>>problematic ones going back to July 2024. Perhaps the version used to
> >>>>process this dataset was one of the unlucky ones.
> >>>>
> >>>> Apologies if this is a red herring, but that check is worth
> >>>>making.
> >>>>
> >>>>
> >>>> Best wishes,
> >>>>
> >>>> Gerard
> >>>>
> >>>>--
> >>>>On Thu, May 29, 2025 at 11:05:52AM +0100, Eleanor Dodson wrote:
> >>>>>After inspecting the log files it is obvious the problem is in the data
> >>>>>integration. (Wilson B negative! highest resolution shell stronger than
> >>>>>low
> >>>>>res, etc..)
> >>>>>I think you used XDS for that? Do you have an alternative set of
> >>>>>integrated
> >>>>>unmerged data, eg done with DIALS?
> >>>>>If so I would use that, or download a new version of XDS and try again..
> >>>>>
> >>>>>On Wed, 28 May 2025 at 07:56, Janani Ganesh <jganesh3...@gmail.com>
> >>>>>wrote:
> >>>>>
> >>>>>>Dear Martin,
> >>>>>>
> >>>>>>Thank you for your suggestion. I will try it out and let you know.
> >>>>>>
> >>>>>>Regards,
> >>>>>>Janani Ganesh
> >>>>>>
> >>>>>>On Tue, 27 May 2025 09:01:06 +0100, Martin Mal�
> >>>>>><martin.maly...@email.cz>
> >>>>>>wrote:
> >>>>>>
> >>>>>>>Dear Janani,
> >>>>>>>
> >>>>>>>There has been recently a significant development in twin refinement
> >>>>>>>algorithms in Servalcat. Could you try this program and let us know if
> >>>>>>>it helps?
> >>>>>>>You can find Servalcat in i2 in the most recent version of CCP4. You
> >>>>>>>can
> >>>>>>>also run it in terminal/CCP4Console:
> >>>>>>>servalcat refine_xtal_norefmac --hklin data.mtz --model model.pdb -s
> >>>>>>>xray --labin I,IMEAN,FreeR_flag --twin
> >>>>>>>(Assuming your MTZ data file has columns I,IMEAN,FreeR_flag.)
> >>>>>>>
> >>>>>>>Full list of options:
> >>>>>>>servalcat refine_xtal_norefmac -h
> >>>>>>>
> >>>>>>>Please feel free to ask if anything was not clear.
> >>>>>>>Cheers,
> >>>>>>>Martin
> >>>>>>>
> >>>>>>>
> >>>>>>>On 27/05/2025 07:00, Janani Ganesh wrote:
> >>>>>>>>Hello,
> >>>>>>>>
> >>>>>>>>I recently collected diffraction data and I was able to do the data
> >>>>>>processing upto 1.88 �. I observed the R-merge remained the same
> >>>>>>throughout
> >>>>>>the resolution bins. I could solve the structure using Phaser in the
> >>>>>>space
> >>>>>>group P32 2 1 (no. 154). But when I refined the structure the R-factors
> >>>>>>remained ~ 35 % even after several cycles of refinement and model
> >>>>>>building.
> >>>>>>>>Then I looked for twinning in my data using Xtriage, Detwin,
> >>>>>>>>Pointless,
> >>>>>>etc which indicated twining. Subsequently, I started twin refinement
> >>>>>>with
> >>>>>>Refmac using twin law (-h, -k, l). The R-factors came down to ~25%.
> >>>>>>However, the B-factors refined to 0.5 for all the atoms. I tried
> >>>>>>redoing
> >>>>>>the data processing at lower symmetry space groups, P3, P31, P32,
> >>>>>>followed
> >>>>>>by twin refinement but the issue with the B-factors persisted.
> >>>>>>>>I would appreciate any suggestions and advice towards resolving this
> >>>>>>problem.
> >>>>>>>>Regards,
> >>>>>>>>Janani Ganesh
> >>>>>>>>
> >>>>>>>>########################################################################
> >>>>>>>>
> >>>>>>>>To unsubscribe from the CCP4BB list, click the following link:
> >>>>>>>>https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB&A=1
> >>>>>>>>
> >>>>>>>>This message was issued to members of www.jiscmail.ac.uk/CCP4BB, a
> >>>>>>mailing list hosted by www.jiscmail.ac.uk, terms & conditions are
> >>>>>>available at https://www.jiscmail.ac.uk/policyandsecurity/
> >>>>>>>########################################################################
> >>>>>>>
> >>>>>>>To unsubscribe from the CCP4BB list, click the following link:
> >>>>>>>https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB&A=1
> >>>>>>>
> >>>>>>>This message was issued to members of www.jiscmail.ac.uk/CCP4BB, a
> >>>>>>mailing list hosted by www.jiscmail.ac.uk, terms & conditions are
> >>>>>>available at https://www.jiscmail.ac.uk/policyandsecurity/
> >>>>>>
> >>>>>>########################################################################
> >>>>>>
> >>>>>>To unsubscribe from the CCP4BB list, click the following link:
> >>>>>>https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB&A=1
> >>>>>>
> >>>>>>This message was issued to members of www.jiscmail.ac.uk/CCP4BB, a
> >>>>>>mailing list hosted by www.jiscmail.ac.uk, terms & conditions are
> >>>>>>available at https://www.jiscmail.ac.uk/policyandsecurity/
> >>>>>>
> >>>>>########################################################################
> >>>>>
> >>>>>To unsubscribe from the CCP4BB list, click the following link:
> >>>>>https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB&A=1
> >>>>>
> >>>>>This message was issued to members of www.jiscmail.ac.uk/CCP4BB, a
> >>>>>mailing list hosted by www.jiscmail.ac.uk, terms & conditions are
> >>>>>available at https://www.jiscmail.ac.uk/policyandsecurity/
> >>>>########################################################################
> >>>>
> >>>>To unsubscribe from the CCP4BB list, click the following link:
> >>>>https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB&A=1
> >>>>
> >>>>This message was issued to members of www.jiscmail.ac.uk/CCP4BB, a
> >>>>mailing list hosted by www.jiscmail.ac.uk, terms & conditions are
> >>>>available at https://www.jiscmail.ac.uk/policyandsecurity/
> >>>########################################################################
> >>>
> >>>To unsubscribe from the CCP4BB list, click the following link:
> >>>https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB&A=1
> >>>
> >>>This message was issued to members of www.jiscmail.ac.uk/CCP4BB, a mailing
> >>>list hosted by www.jiscmail.ac.uk, terms & conditions are available at
> >>>https://www.jiscmail.ac.uk/policyandsecurity/
> >>
> >>
> >>########################################################################
> >>
> >>To unsubscribe from the CCP4BB list, click the following link:
> >>https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB&A=1
> >>
> >>This message was issued to members of www.jiscmail.ac.uk/CCP4BB, a mailing
> >>list hosted by www.jiscmail.ac.uk, terms & conditions are available at
> >>https://www.jiscmail.ac.uk/policyandsecurity/
> >########################################################################
> >
> >To unsubscribe from the CCP4BB list, click the following link:
> >https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB&A=1
> >
> >This message was issued to members of www.jiscmail.ac.uk/CCP4BB, a mailing
> >list hosted by www.jiscmail.ac.uk, terms & conditions are available at
> >https://www.jiscmail.ac.uk/policyandsecurity/
>
> --
> Engin Özkan, Ph.D.
> Associate Professor
> Dept of Biochemistry and Molecular Biology
> University of Chicago
> Phone: (773) 834-5498
> http://ozkan.uchicago.edu
########################################################################
To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB&A=1
This message was issued to members of www.jiscmail.ac.uk/CCP4BB, a mailing list
hosted by www.jiscmail.ac.uk, terms & conditions are available at
https://www.jiscmail.ac.uk/policyandsecurity/
