How about including all of this stuff in refinement? Only adds another ~10-100 [restrained] parameters…
JPK +++++++++++++++++++++++++++++++++++++++++++++++++ Jacob Pearson Keller Research Scientist / Looger Lab HHMI Janelia Research Campus 19700 Helix Dr, Ashburn, VA 20147 Desk: (571)209-4000 x3159 Cell: (301)592-7004 +++++++++++++++++++++++++++++++++++++++++++++++++ The content of this email is confidential and intended for the recipient specified in message only. It is strictly forbidden to share any part of this message with any third party, without a written consent of the sender. If you received this message by mistake, please reply to this message and follow with its deletion, so that we can ensure such a mistake does not occur in the future. From: CCP4 bulletin board [mailto:CCP4BB@JISCMAIL.AC.UK] On Behalf Of wtempel Sent: Wednesday, January 24, 2018 12:19 PM To: CCP4BB@JISCMAIL.AC.UK Subject: Re: [ccp4bb] Issues with latest XDS (20171218) I agree that accurate and complete image header information is useful, but not universal. How about a routine similar to dials.discover_better_experimental_model? W. On Wed, Jan 24, 2018 at 10:12 AM, Engin Özkan <eoz...@uchicago.edu<mailto:eoz...@uchicago.edu>> wrote: Dear all, I have to agree with all sentiments stated, and I have definitely seen the value of restraining distance values in weak and anisotropic data, which is great for those of us who deal with such data on a regular basis. However, I also just went through a search of all refined DISTANCE values on all (~20) CORRECT.LP files on my hard drive. Mismatches between input and refined values of up to 0.25% (1 mm in 400 mm) is not uncommon for data collected over several different beam lines. Many of us collect data at synchrotrons remotely and/or without ever talking to a beamline scientists, and expecting users to end up with perfect distance values in their auto-scripted XDS.INP files might be expecting the perfect in an imperfect, fast-moving data-collection world. A solution that accommodates both viewpoints, if at all possible, would indeed be great. Best, Engin On 1/24/18 8:06 AM, "Weiergräber, Oliver H." wrote: Dear Clemens, dear Kay, I absolutely agree with your statement regarding responsibilities. Unfortunately, for a user of a synchrotron beamline, it is hardly possible to _know_ that the distance (or any beam or camera parameter) provided by the beamline software is in error. In the end indications can only be derived from the behaviour of software used for processing. Of course developers are not responsible for hardware issues, but since such issues do occur in real life, the software may still help spotting them ;-) Given that IDXREF runs very fast, why not have it probe the shifts of parameters for a full refinement scope and issue a warning if things look suspicious, giving the user options how to proceed (similar to the not-so-infrequent case of "insufficient percentage of indexed reflections"). Best regards Oliver ================================================ PD Dr. Oliver H. Weiergräber Institute of Complex Systems ICS-6: Structural Biochemistry Tel.: +49 2461 61-2028<tel:%2B49%202461%2061-2028> Fax: +49 2461 61-9540<tel:%2B49%202461%2061-9540> ================================================ ________________________________________ From: CCP4 bulletin board [CCP4BB@JISCMAIL.AC.UK<mailto:CCP4BB@JISCMAIL.AC.UK>] on behalf of Clemens Vonrhein [vonrh...@globalphasing.com<mailto:vonrh...@globalphasing.com>] Sent: Wednesday, January 24, 2018 2:39 PM To: CCP4BB@JISCMAIL.AC.UK<mailto:CCP4BB@JISCMAIL.AC.UK> Subject: Re: [ccp4bb] Issues with latest XDS (20171218) Dear Oliver, yes, there are other changes to the parameter refinement procedure within XDS as far as I understand. These were introduced to robustify XDS processing (and parameter refinement) when encountering very poor datasets, cases where the crystal moves out of the beam in certain orientations or empty loops. That works very well it seems - at least for us (and we were involved in discussing those cases with both Wolfgang Kabsch and Kay Diederichs). I'm sure you agree that if the crystal to detector distance in the header is wrong by such a large amount (over 1 mm), the (proper) solution is to fix this at the point of collecting the data and writing the header. As Kay says: there should be no reason for an instrument/beamline to not know that distance very accurately and to write the correct value in the header. It has the same importance as e.g. wavelength/energy or oscillation range. Of course, if a user already has images with an inaccurate value and wants to process those, the old behaviour of XDS might be able to fix that particular problem for you. But this does feel like patching up simple-to-fix problems at the wrong stage, namely processing instead of at the beamline side ... with the "danger" that they will never get fixed properly because processing software will "somehow patch things up anyway". All those software packages do already a very large amount of workarounds because of real life being what it is, but if we want to achieve the highest quality of processed data I think we can expect the same amount of accuracy when it comes to the description of the experiment that goes into programs like XDS. I think the current XDS behaviour is much more robust and reliable _if_ the experiment is described accurately. The latter is the responsibility of the beamline (image headers etc) and the user. It should not be XDS's task to calibrate the instrument parameters (against a single sweep dataset from whatever crystal it sees), but rather index, integrate and scale/merge the data collected and described. Anyway, just my 2 cents ;-) Cheers Clemens On Wed, Jan 24, 2018 at 09:36:19AM +0000, "Weiergräber, Oliver H." wrote: a) Recycling of geometry parameters is certainly worth trying, although in our experience it rarely yields large improvements. Obviously, XDS used to do a pretty good job in default mode. In the latest version, however, since refinement of the detector distance is disabled by default, recycling cannot be expected to (and indeed does not) help. b) The data I mentioned have been collected at ESRF ID29 and processed using the XDS.INP file they provided, with slight modifications unrelated to geometry refinement. c) Yes, defining REFINE(IDXREF) to include POSITION does not solve the problem. It seems that refinement of the distance is still strongly restrained, so there must have been changes to the code other than removal of POSITION refinement from the REFINE(IDXREF) defaults. Consequently, parameter recycling does not help much even with POSITION refinement re-enabled in IDXREF. The bottom line is that there appears to be no obvious way to restore the previous behaviour by just changing parameters (but of course I may have missed something). I think such an option is urgently required. The most worrisome aspect about this issue, in my opinion, is that it may go unnoticed in many cases. The way it stands now, people affected by inaccuracies in detector distance (or maybe other parameters) may be misled to choose cut-offs for integration at much too high dmin, discarding valuable data. I am happy to provide data needed for investigation. Let's discuss this part off-list. Best regards Oliver ================================================ PD Dr. Oliver H. Weiergräber Institute of Complex Systems ICS-6: Structural Biochemistry Tel.: +49 2461 61-2028<tel:%2B49%202461%2061-2028> Fax: +49 2461 61-9540<tel:%2B49%202461%2061-9540> ================================================ ________________________________________ From: CCP4 bulletin board [CCP4BB@JISCMAIL.AC.UK<mailto:CCP4BB@JISCMAIL.AC.UK>] on behalf of Kay Diederichs [kay.diederi...@uni-konstanz.de<mailto:kay.diederi...@uni-konstanz.de>] Sent: Tuesday, January 23, 2018 9:16 PM To: CCP4BB@JISCMAIL.AC.UK<mailto:CCP4BB@JISCMAIL.AC.UK> Subject: Re: [ccp4bb] Issues with latest XDS (20171218) Dear Oliver, sorry for the trouble! A default should be correct in the majority of situations, but it's impossible to have it work for _all_ situations. The XDS default for REFINE(IDXREF) was changed (i.e. POSITION was removed) to improve the indexing for weak and lousy data, _and_ because the distance values from the header are nowadays almost always accurate. For data with very high resolution such as yours, and significantly wrong distance, this means that at high resolution in particular this default will lead to worse data. generate_XDS.INP (in XDSwiki) and (I think) autoPROC and xia2 explicitly include POSITION in the REFINE(IDXREF), i.e. override the default. Where did you get XDS.INP from? Some comments: a) this is why I recommend, for data sets that may be important, to always do one cycle of optimization, i.e. after CORRECT, "mv GXPARM.XDS XPARM.XDS", change XDS.INP to have the correct high-resolution cutoff (cutting after the last shell which still has a "*" in the CC1/2 column), adjust the FRIEDEL'S_LAW= setting, and run JOB=INTEGRATE CORRECT. In your case this would make the refined distance available to INTEGRATE, and should result in very good data. b) at which beamline did you collect the data? Such a high difference between refined distance and distance from header is unusual, and should be reported to (and fixed by) the beamline staff. c) for this beamline at least, one should use REFINE(IDXREF)=CELL BEAM ORIENTATION AXIS POSITION . I understand that you tried this and it didn't work? Maybe there is something else wrong then, e.g. another line later in XDS.INP resetting REFINE(IDXREF) to something else. If you cannot find the reason why including POSITiON into REFINE(IDXREF) does not help, pls send me enough data to reproduce the problem. best wishes, Kay On Tue, 23 Jan 2018 14:31:09 +0000, "Weiergräber, Oliver H." <o.h.weiergrae...@fz-juelich.de<mailto:o.h.weiergrae...@fz-juelich.de>> wrote: Dear all, After upgrading XDS from build date 20170601 to 20171218, I am experiencing severe degradation of apparent data quality reported by CORRECT for certain data sets. Following first indications of issues with a slightly problematic candidate, I went back to a previously very well-behaved test case with diffraction extending beyond 1.1 A. Using the same input with both program versions, statistics are not too different out to approx. 1.6 A, but become more and more divergent in outer shells. These are the values for the highest-resolution shell (1.10--1.16 A): 20170601: I/sigI = 1.80; Rmeas = 59.9 %; CC(1/2) = 82.0 % 20171218: I/sigI = 0.14; Rmeas = 506.4 %; CC(1/2) = 5.8 % The refined cell constants are unreasonably different as well. Obviously, something is going awfully wrong here, presumably related to errors in geometry parameters (which are expected to become increasingly detrimental with decreasing dmin). As it turns out, geometry refinement behaves differently in the latest version of XDS: most notably, IDXREF no longer refines the detector distance by default. This is significant in this case, as in version 20170601 the distance would shift by as much as 1.3 mm, resulting in successful integration and scaling. Unfortunately, re-adding POSITION refinement into REFINE(IDXREF) does not help, and even having it in all refinement scopes (IDXREF, INTEGRATE, CORRECT) only yields a limited improvement of CORRECT statistics. It is worth noting that examination of a dataset from an unrelated crystal (dmin = 1.4 A) did not reveal such enormous differences -- in this case, however, the shift in detector distance applied in the old version of XDS was quite small (0.2 mm), which, together with the lower resolution, explains the absence of large discrepancies. To sum up, I suspect that, due to recent changes to the XDS code, the refinement of geometry parameters is now overly restrained, resulting in drastic problems in cases where the detector distance is not as precise as desirable (and even more so at very high resolution). For such datasets, the new version appears to be essentially unusable (at least within the parameter space I have tested), and even in modest cases may often be inferior to the previous one. Is there an option to revert to the former behaviour? Best regards Oliver ================================================ PD Dr. Oliver H. Weiergr�ber Institute of Complex Systems ICS-6: Structural Biochemistry Tel.: +49 2461 61-2028<tel:%2B49%202461%2061-2028> Fax: +49 2461 61-9540<tel:%2B49%202461%2061-9540> ================================================ ------------------------------------------------------------------------------------------------ ------------------------------------------------------------------------------------------------ Forschungszentrum Juelich GmbH 52425 Juelich Sitz der Gesellschaft: Juelich Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender), Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt, Prof. Dr. Sebastian M. Schmidt ------------------------------------------------------------------------------------------------ ------------------------------------------------------------------------------------------------ -- *-------------------------------------------------------------- * Clemens Vonrhein, Ph.D. vonrhein AT GlobalPhasing DOT com * Global Phasing Ltd., Sheraton House, Castle Park * Cambridge CB3 0AX, UK www.globalphasing.com<http://www.globalphasing.com> *-------------------------------------------------------------- -- *-------------------------------------------------------------- * Clemens Vonrhein, Ph.D. vonrhein AT GlobalPhasing DOT com * Global Phasing Ltd., Sheraton House, Castle Park * Cambridge CB3 0AX, UK www.globalphasing.com<http://www.globalphasing.com> *--------------------------------------------------------------
