Dear Phil,
I am not sure that I follow you there: perfectly twinned acentric
intensities will follow a Chi-squared distribution with four degrees
of freedom, while untwinned ones have only two degrees of freedom. The
higher the number of degrees of freedom, the more the Chi-squared
distribution will peak around its mean value (Central Limit Theorem).
This trend is clearly visible in going from the red to the blue curve.
The green curve shows an even stronger trend towards a peaked
distribution, so actually the data coming our of 20240712 are, so to
speak, "hyper-twinned". This shift away from low values is consistent
with Gleb Bourenkov's findings, obtained by integrating images with
very low signal created with our image simulator, that integrated
intensities produced by the six successive buggy versions of XDS since
July last year (20240712, 20240723, 20241002, 20250119, 20250224 and
20250320) were affected by biases, mostly positive and causing a
depletion of the weak intensity population (hence spurious twinning
diagnostics), plus an overestimation of I/sig(I) and even Wilson plots
curving upwards (!) at high resolution.
Please peruse the autoPROC Wiki page for which I sent a link
earlier and take a look at the plots it contains: it is all described
in there.
With best wishes,
Gerard.
--
On Thu, May 29, 2025 at 05:30:54PM +0100, Phil Evans wrote:
> Dear Kay
>
> This is a plot I sent to Eleanor yesterday, from her log file. The cumulative
> intensity plot shows I think that there are too few weak intensities, the
> opposite of twinning. I don;t know where it comes from but it’s not right!
> Best
> Phil
>
> ########################################################################
>
> To unsubscribe from the CCP4BB list, click the following link:
> https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB&A=1
>
> This message was issued to members of www.jiscmail.ac.uk/CCP4BB, a mailing
> list hosted by www.jiscmail.ac.uk, terms & conditions are available at
> https://www.jiscmail.ac.uk/policyandsecurity/
>
>
>
> > On 29 May 2025, at 12:03, Kay Diederichs <kay.diederi...@uni-konstanz.de>
> > wrote:
> >
> > CAUTION: This email originated from outside of the LMB:
> > .-owner-ccp...@jiscmail.ac.uk-.
> > Do not click links or open attachments unless you recognize the sender and
> > know the content is safe.
> > If you think this is a phishing email, please forward it to
> > phish...@mrc-lmb.cam.ac.uk
> > --
> >
> > Dear Gerard,
> >
> > part of the problem was probably indeed due to the use of XDS
> > BUILT=20240712 (that version was used for CORRECT.LP that Janani Ganesh
> > sent to me). I obtained the raw data from Janani Ganesh and processed with
> > both the current XDS (BUILT=20250430), and the bad version from July last
> > year. My findings:
> > - with the bad version the data appear twinned, whereas with the current
> > version the data appear untwinned
> > - Wilson B is 34 A^2 with the current version, and significantly (and
> > unrealistically) lower with the bad version
> >
> > The structure can be satisfactorily solved and refined with data from the
> > current XDS version. With data from the bad version, average B goes down to
> > 19 and the R-values stall at high values.
> >
> > This does not explain the weird B-values that were reported, though. So I
> > suspect another problem.
> >
> > Best,
> > Kay
> >
> >
> >
> > On Thu, 29 May 2025 11:37:14 +0100, Gerard Bricogne
> > <g...@globalphasing.com> wrote:
> >
> >> Dear all,
> >>
> >> I wrote yesterday a contribution to this thread, but mistakenly
> >> used a simple Reply instead of a Group Reply, so that my message did
> >> not reach the BB while it intended to, as shown by the introductory
> >> "Dear all".
> >>
> >> I am including it below, as it contains a link to some material
> >> that is of general interest but that we had not yet found a way of
> >> disseminating to the whole community. Users of XDS, or of XDS-based
> >> pipelines, please take notice and do take a close look at this link,
> >> and ultimately at the BUILT= information about the exact XDS version
> >> used in producing your results - it can make a lot of difference.
> >>
> >>
> >> With best wishes,
> >>
> >> Gerard.
> >>
> >> --
> >> Date: Wed, 28 May 2025 16:18:25 +0100
> >> From: Gerard Bricogne <gb10>
> >> Subject: Re: [ccp4bb] Requesting help with twin refinement
> >> To: Eleanor Dodson <eleanor.dod...@york.ac.uk>
> >>
> >> Dear all,
> >>
> >> These kinds of anomalies (overoptimistic estimates of resolution,
> >> unphysical persistence of signal at high resolution, spurious twinning
> >> diagnostics) are reminiscent of what we observed with some of the
> >> recent versions of XDS, as documented at
> >>
> >> https://www.globalphasing.com/autoproc/wiki/index.cgi?ComparisonProcessing202504
> >>
> >> and three predecessor pages referenced from within this one.
> >>
> >> Only one version is currently available from the XDS download
> >> page (that we consider as being OK) but there was a succession of many
> >> problematic ones going back to July 2024. Perhaps the version used to
> >> process this dataset was one of the unlucky ones.
> >>
> >> Apologies if this is a red herring, but that check is worth
> >> making.
> >>
> >>
> >> Best wishes,
> >>
> >> Gerard
> >>
> >> --
> >> On Thu, May 29, 2025 at 11:05:52AM +0100, Eleanor Dodson wrote:
> >>> After inspecting the log files it is obvious the problem is in the data
> >>> integration. (Wilson B negative! highest resolution shell stronger than
> >>> low
> >>> res, etc..)
> >>> I think you used XDS for that? Do you have an alternative set of
> >>> integrated
> >>> unmerged data, eg done with DIALS?
> >>> If so I would use that, or download a new version of XDS and try again..
> >>>
> >>> On Wed, 28 May 2025 at 07:56, Janani Ganesh <jganesh3...@gmail.com> wrote:
> >>>
> >>>> Dear Martin,
> >>>>
> >>>> Thank you for your suggestion. I will try it out and let you know.
> >>>>
> >>>> Regards,
> >>>> Janani Ganesh
> >>>>
> >>>> On Tue, 27 May 2025 09:01:06 +0100, Martin Mal� <martin.maly...@email.cz>
> >>>> wrote:
> >>>>
> >>>>> Dear Janani,
> >>>>>
> >>>>> There has been recently a significant development in twin refinement
> >>>>> algorithms in Servalcat. Could you try this program and let us know if
> >>>>> it helps?
> >>>>> You can find Servalcat in i2 in the most recent version of CCP4. You can
> >>>>> also run it in terminal/CCP4Console:
> >>>>> servalcat refine_xtal_norefmac --hklin data.mtz --model model.pdb -s
> >>>>> xray --labin I,IMEAN,FreeR_flag --twin
> >>>>> (Assuming your MTZ data file has columns I,IMEAN,FreeR_flag.)
> >>>>>
> >>>>> Full list of options:
> >>>>> servalcat refine_xtal_norefmac -h
> >>>>>
> >>>>> Please feel free to ask if anything was not clear.
> >>>>> Cheers,
> >>>>> Martin
> >>>>>
> >>>>>
> >>>>> On 27/05/2025 07:00, Janani Ganesh wrote:
> >>>>>> Hello,
> >>>>>>
> >>>>>> I recently collected diffraction data and I was able to do the data
> >>>> processing upto 1.88 �. I observed the R-merge remained the same
> >>>> throughout
> >>>> the resolution bins. I could solve the structure using Phaser in the
> >>>> space
> >>>> group P32 2 1 (no. 154). But when I refined the structure the R-factors
> >>>> remained ~ 35 % even after several cycles of refinement and model
> >>>> building.
> >>>>>>
> >>>>>> Then I looked for twinning in my data using Xtriage, Detwin, Pointless,
> >>>> etc which indicated twining. Subsequently, I started twin refinement with
> >>>> Refmac using twin law (-h, -k, l). The R-factors came down to ~25%.
> >>>> However, the B-factors refined to 0.5 for all the atoms. I tried redoing
> >>>> the data processing at lower symmetry space groups, P3, P31, P32,
> >>>> followed
> >>>> by twin refinement but the issue with the B-factors persisted.
> >>>>>>
> >>>>>> I would appreciate any suggestions and advice towards resolving this
> >>>> problem.
> >>>>>>
> >>>>>> Regards,
> >>>>>> Janani Ganesh
> >>>>>>
> >>>>>> ########################################################################
> >>>>>>
> >>>>>> To unsubscribe from the CCP4BB list, click the following link:
> >>>>>> https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB&A=1
> >>>>>>
> >>>>>> This message was issued to members of www.jiscmail.ac.uk/CCP4BB, a
> >>>> mailing list hosted by www.jiscmail.ac.uk, terms & conditions are
> >>>> available at https://www.jiscmail.ac.uk/policyandsecurity/
> >>>>>
> >>>>> ########################################################################
> >>>>>
> >>>>> To unsubscribe from the CCP4BB list, click the following link:
> >>>>> https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB&A=1
> >>>>>
> >>>>> This message was issued to members of www.jiscmail.ac.uk/CCP4BB, a
> >>>> mailing list hosted by www.jiscmail.ac.uk, terms & conditions are
> >>>> available at https://www.jiscmail.ac.uk/policyandsecurity/
> >>>>
> >>>> ########################################################################
> >>>>
> >>>> To unsubscribe from the CCP4BB list, click the following link:
> >>>> https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB&A=1
> >>>>
> >>>> This message was issued to members of www.jiscmail.ac.uk/CCP4BB, a
> >>>> mailing list hosted by www.jiscmail.ac.uk, terms & conditions are
> >>>> available at https://www.jiscmail.ac.uk/policyandsecurity/
> >>>>
> >>>
> >>> ########################################################################
> >>>
> >>> To unsubscribe from the CCP4BB list, click the following link:
> >>> https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB&A=1
> >>>
> >>> This message was issued to members of www.jiscmail.ac.uk/CCP4BB, a
> >>> mailing list hosted by www.jiscmail.ac.uk, terms & conditions are
> >>> available at https://www.jiscmail.ac.uk/policyandsecurity/
> >>
> >> ########################################################################
> >>
> >> To unsubscribe from the CCP4BB list, click the following link:
> >> https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB&A=1
> >>
> >> This message was issued to members of www.jiscmail.ac.uk/CCP4BB, a mailing
> >> list hosted by www.jiscmail.ac.uk, terms & conditions are available at
> >> https://www.jiscmail.ac.uk/policyandsecurity/
> >
> > ########################################################################
> >
> > To unsubscribe from the CCP4BB list, click the following link:
> > https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB&A=1
> >
> > This message was issued to members of www.jiscmail.ac.uk/CCP4BB, a mailing
> > list hosted by www.jiscmail.ac.uk, terms & conditions are available at
> > https://www.jiscmail.ac.uk/policyandsecurity/
>
>
>
> ########################################################################
>
> To unsubscribe from the CCP4BB list, click the following link:
> https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB&A=1
>
> This message was issued to members of www.jiscmail.ac.uk/CCP4BB, a mailing
> list hosted by www.jiscmail.ac.uk, terms & conditions are available at
> https://www.jiscmail.ac.uk/policyandsecurity/
########################################################################
To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB&A=1
This message was issued to members of www.jiscmail.ac.uk/CCP4BB, a mailing list
hosted by www.jiscmail.ac.uk, terms & conditions are available at
https://www.jiscmail.ac.uk/policyandsecurity/
