Gerard, of course you are right and I was confused. Twinning causes too few weak intensities and too few strong ones, ie reduced variance of the distribution.
But this distribution is certainly wrong! Phil > On 29 May 2025, at 18:06, Gerard Bricogne > <0000ead34ec923cf-dmarc-requ...@jiscmail.ac.uk> wrote: > > CAUTION: This email originated from outside of the LMB: > .-owner-ccp...@jiscmail.ac.uk-. > Do not click links or open attachments unless you recognize the sender and > know the content is safe. > If you think this is a phishing email, please forward it to > phish...@mrc-lmb.cam.ac.uk > -- > > Dear Phil, > > I am not sure that I follow you there: perfectly twinned acentric > intensities will follow a Chi-squared distribution with four degrees > of freedom, while untwinned ones have only two degrees of freedom. The > higher the number of degrees of freedom, the more the Chi-squared > distribution will peak around its mean value (Central Limit Theorem). > This trend is clearly visible in going from the red to the blue curve. > > The green curve shows an even stronger trend towards a peaked > distribution, so actually the data coming our of 20240712 are, so to > speak, "hyper-twinned". This shift away from low values is consistent > with Gleb Bourenkov's findings, obtained by integrating images with > very low signal created with our image simulator, that integrated > intensities produced by the six successive buggy versions of XDS since > July last year (20240712, 20240723, 20241002, 20250119, 20250224 and > 20250320) were affected by biases, mostly positive and causing a > depletion of the weak intensity population (hence spurious twinning > diagnostics), plus an overestimation of I/sig(I) and even Wilson plots > curving upwards (!) at high resolution. > > Please peruse the autoPROC Wiki page for which I sent a link > earlier and take a look at the plots it contains: it is all described > in there. > > > With best wishes, > > Gerard. > > -- > On Thu, May 29, 2025 at 05:30:54PM +0100, Phil Evans wrote: >> Dear Kay >> >> This is a plot I sent to Eleanor yesterday, from her log file. The >> cumulative intensity plot shows I think that there are too few weak >> intensities, the opposite of twinning. I don;t know where it comes from but >> it’s not right! >> Best >> Phil >> >> ######################################################################## >> >> To unsubscribe from the CCP4BB list, click the following link: >> https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB&A=1 >> >> This message was issued to members of www.jiscmail.ac.uk/CCP4BB, a mailing >> list hosted by www.jiscmail.ac.uk, terms & conditions are available at >> https://www.jiscmail.ac.uk/policyandsecurity/ > > >> >> >> >>> On 29 May 2025, at 12:03, Kay Diederichs <kay.diederi...@uni-konstanz.de> >>> wrote: >>> >>> CAUTION: This email originated from outside of the LMB: >>> .-owner-ccp...@jiscmail.ac.uk-. >>> Do not click links or open attachments unless you recognize the sender and >>> know the content is safe. >>> If you think this is a phishing email, please forward it to >>> phish...@mrc-lmb.cam.ac.uk >>> -- >>> >>> Dear Gerard, >>> >>> part of the problem was probably indeed due to the use of XDS >>> BUILT=20240712 (that version was used for CORRECT.LP that Janani Ganesh >>> sent to me). I obtained the raw data from Janani Ganesh and processed with >>> both the current XDS (BUILT=20250430), and the bad version from July last >>> year. My findings: >>> - with the bad version the data appear twinned, whereas with the current >>> version the data appear untwinned >>> - Wilson B is 34 A^2 with the current version, and significantly (and >>> unrealistically) lower with the bad version >>> >>> The structure can be satisfactorily solved and refined with data from the >>> current XDS version. With data from the bad version, average B goes down to >>> 19 and the R-values stall at high values. >>> >>> This does not explain the weird B-values that were reported, though. So I >>> suspect another problem. >>> >>> Best, >>> Kay >>> >>> >>> >>> On Thu, 29 May 2025 11:37:14 +0100, Gerard Bricogne >>> <g...@globalphasing.com> wrote: >>> >>>> Dear all, >>>> >>>> I wrote yesterday a contribution to this thread, but mistakenly >>>> used a simple Reply instead of a Group Reply, so that my message did >>>> not reach the BB while it intended to, as shown by the introductory >>>> "Dear all". >>>> >>>> I am including it below, as it contains a link to some material >>>> that is of general interest but that we had not yet found a way of >>>> disseminating to the whole community. Users of XDS, or of XDS-based >>>> pipelines, please take notice and do take a close look at this link, >>>> and ultimately at the BUILT= information about the exact XDS version >>>> used in producing your results - it can make a lot of difference. >>>> >>>> >>>> With best wishes, >>>> >>>> Gerard. >>>> >>>> -- >>>> Date: Wed, 28 May 2025 16:18:25 +0100 >>>> From: Gerard Bricogne <gb10> >>>> Subject: Re: [ccp4bb] Requesting help with twin refinement >>>> To: Eleanor Dodson <eleanor.dod...@york.ac.uk> >>>> >>>> Dear all, >>>> >>>> These kinds of anomalies (overoptimistic estimates of resolution, >>>> unphysical persistence of signal at high resolution, spurious twinning >>>> diagnostics) are reminiscent of what we observed with some of the >>>> recent versions of XDS, as documented at >>>> >>>> https://www.globalphasing.com/autoproc/wiki/index.cgi?ComparisonProcessing202504 >>>> >>>> and three predecessor pages referenced from within this one. >>>> >>>> Only one version is currently available from the XDS download >>>> page (that we consider as being OK) but there was a succession of many >>>> problematic ones going back to July 2024. Perhaps the version used to >>>> process this dataset was one of the unlucky ones. >>>> >>>> Apologies if this is a red herring, but that check is worth >>>> making. >>>> >>>> >>>> Best wishes, >>>> >>>> Gerard >>>> >>>> -- >>>> On Thu, May 29, 2025 at 11:05:52AM +0100, Eleanor Dodson wrote: >>>>> After inspecting the log files it is obvious the problem is in the data >>>>> integration. (Wilson B negative! highest resolution shell stronger than >>>>> low >>>>> res, etc..) >>>>> I think you used XDS for that? Do you have an alternative set of >>>>> integrated >>>>> unmerged data, eg done with DIALS? >>>>> If so I would use that, or download a new version of XDS and try again.. >>>>> >>>>> On Wed, 28 May 2025 at 07:56, Janani Ganesh <jganesh3...@gmail.com> wrote: >>>>> >>>>>> Dear Martin, >>>>>> >>>>>> Thank you for your suggestion. I will try it out and let you know. >>>>>> >>>>>> Regards, >>>>>> Janani Ganesh >>>>>> >>>>>> On Tue, 27 May 2025 09:01:06 +0100, Martin Mal� <martin.maly...@email.cz> >>>>>> wrote: >>>>>> >>>>>>> Dear Janani, >>>>>>> >>>>>>> There has been recently a significant development in twin refinement >>>>>>> algorithms in Servalcat. Could you try this program and let us know if >>>>>>> it helps? >>>>>>> You can find Servalcat in i2 in the most recent version of CCP4. You can >>>>>>> also run it in terminal/CCP4Console: >>>>>>> servalcat refine_xtal_norefmac --hklin data.mtz --model model.pdb -s >>>>>>> xray --labin I,IMEAN,FreeR_flag --twin >>>>>>> (Assuming your MTZ data file has columns I,IMEAN,FreeR_flag.) >>>>>>> >>>>>>> Full list of options: >>>>>>> servalcat refine_xtal_norefmac -h >>>>>>> >>>>>>> Please feel free to ask if anything was not clear. >>>>>>> Cheers, >>>>>>> Martin >>>>>>> >>>>>>> >>>>>>> On 27/05/2025 07:00, Janani Ganesh wrote: >>>>>>>> Hello, >>>>>>>> >>>>>>>> I recently collected diffraction data and I was able to do the data >>>>>> processing upto 1.88 �. I observed the R-merge remained the same >>>>>> throughout >>>>>> the resolution bins. I could solve the structure using Phaser in the >>>>>> space >>>>>> group P32 2 1 (no. 154). But when I refined the structure the R-factors >>>>>> remained ~ 35 % even after several cycles of refinement and model >>>>>> building. >>>>>>>> >>>>>>>> Then I looked for twinning in my data using Xtriage, Detwin, Pointless, >>>>>> etc which indicated twining. Subsequently, I started twin refinement with >>>>>> Refmac using twin law (-h, -k, l). The R-factors came down to ~25%. >>>>>> However, the B-factors refined to 0.5 for all the atoms. I tried redoing >>>>>> the data processing at lower symmetry space groups, P3, P31, P32, >>>>>> followed >>>>>> by twin refinement but the issue with the B-factors persisted. >>>>>>>> >>>>>>>> I would appreciate any suggestions and advice towards resolving this >>>>>> problem. >>>>>>>> >>>>>>>> Regards, >>>>>>>> Janani Ganesh >>>>>>>> >>>>>>>> ######################################################################## >>>>>>>> >>>>>>>> To unsubscribe from the CCP4BB list, click the following link: >>>>>>>> https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB&A=1 >>>>>>>> >>>>>>>> This message was issued to members of www.jiscmail.ac.uk/CCP4BB, a >>>>>> mailing list hosted by www.jiscmail.ac.uk, terms & conditions are >>>>>> available at https://www.jiscmail.ac.uk/policyandsecurity/ >>>>>>> >>>>>>> ######################################################################## >>>>>>> >>>>>>> To unsubscribe from the CCP4BB list, click the following link: >>>>>>> https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB&A=1 >>>>>>> >>>>>>> This message was issued to members of www.jiscmail.ac.uk/CCP4BB, a >>>>>> mailing list hosted by www.jiscmail.ac.uk, terms & conditions are >>>>>> available at https://www.jiscmail.ac.uk/policyandsecurity/ >>>>>> >>>>>> ######################################################################## >>>>>> >>>>>> To unsubscribe from the CCP4BB list, click the following link: >>>>>> https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB&A=1 >>>>>> >>>>>> This message was issued to members of www.jiscmail.ac.uk/CCP4BB, a >>>>>> mailing list hosted by www.jiscmail.ac.uk, terms & conditions are >>>>>> available at https://www.jiscmail.ac.uk/policyandsecurity/ >>>>>> >>>>> >>>>> ######################################################################## >>>>> >>>>> To unsubscribe from the CCP4BB list, click the following link: >>>>> https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB&A=1 >>>>> >>>>> This message was issued to members of www.jiscmail.ac.uk/CCP4BB, a >>>>> mailing list hosted by www.jiscmail.ac.uk, terms & conditions are >>>>> available at https://www.jiscmail.ac.uk/policyandsecurity/ >>>> >>>> ######################################################################## >>>> >>>> To unsubscribe from the CCP4BB list, click the following link: >>>> https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB&A=1 >>>> >>>> This message was issued to members of www.jiscmail.ac.uk/CCP4BB, a mailing >>>> list hosted by www.jiscmail.ac.uk, terms & conditions are available at >>>> https://www.jiscmail.ac.uk/policyandsecurity/ >>> >>> ######################################################################## >>> >>> To unsubscribe from the CCP4BB list, click the following link: >>> https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB&A=1 >>> >>> This message was issued to members of www.jiscmail.ac.uk/CCP4BB, a mailing >>> list hosted by www.jiscmail.ac.uk, terms & conditions are available at >>> https://www.jiscmail.ac.uk/policyandsecurity/ >> >> >> >> ######################################################################## >> >> To unsubscribe from the CCP4BB list, click the following link: >> https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB&A=1 >> >> This message was issued to members of www.jiscmail.ac.uk/CCP4BB, a mailing >> list hosted by www.jiscmail.ac.uk, terms & conditions are available at >> https://www.jiscmail.ac.uk/policyandsecurity/ > > ######################################################################## > > To unsubscribe from the CCP4BB list, click the following link: > https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB&A=1 > > This message was issued to members of www.jiscmail.ac.uk/CCP4BB, a mailing > list hosted by www.jiscmail.ac.uk, terms & conditions are available at > https://www.jiscmail.ac.uk/policyandsecurity/ ######################################################################## To unsubscribe from the CCP4BB list, click the following link: https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB&A=1 This message was issued to members of www.jiscmail.ac.uk/CCP4BB, a mailing list hosted by www.jiscmail.ac.uk, terms & conditions are available at https://www.jiscmail.ac.uk/policyandsecurity/
