Hi Simon, When data are processed with XDS and converted to an mtz file by the route you suggest unmeasured reflections will simply be absent from the mtz file rather than denoted by a missing number flag. Prior to missing number flags (back in CCP4 3.something) it was common practice to leave out missing (unmeasured) terms when calculating 2fo-fc maps etc. This is equivalent to saying that the corresponding structure factor is zero which is an incorrect assumption. Errors will be introduced resulting in noisier maps. Maps are improved if 2fo-fc (or equivalent) is replaced with fc for missing reflections rather than simply assuming they are zero (although I guess this will increase model bias). I believe that CCP4 programs only do this if missing number flags are present in the mtz file (perhaps someone could confirm this). When I convert from XDS to an mtz file I usually use CAD to combine my XDS data with complete list of reflections generated with UNIQUE in CCP4. This ensures that any unmeasured reflections are denoted by a missing number flag. As someone else has mentioned it is good practice to use the UNIQUIFY script at the start of any new project to define a free R set for the rest of the project, including mutant data sets in the same space group etc. I'm not sure how missing data are handled in refinement. Alun. ________________________________
From: CCP4 bulletin board [mailto:[EMAIL PROTECTED] On Behalf Of Simon Kolstoe Sent: 21 February 2008 16:58 To: CCP4BB@JISCMAIL.AC.UK Subject: Re: [ccp4bb] XDS and overlaps In the past I have used the USF program DATAMAN to pick my Rfree in thin shells. Thus my data goes XDS-> XDSCONV-> DATAMAN-> F2MTZ in order to get a reflection file that refmac and phaser are happy with. Do I then need to run the UNIQUEIFY script selecting the option "keep existing FreeR data" in order to avoid the reciprocal asymmetric unit problem? If so this problem must only occur occasionally as I have had structures refining with R & Rfree's in the very low 20s without performing the UNIQUEIFY step. Simon On 21 Feb 2008, at 15:32, Dirk Kostrewa wrote: Usually, I run CAD first after F2MTZ to make sure that the reflections are in the correct reciprocal asymmetric unit for CCP4 programs. I think, UNIQUE on its own doesn't do this, but the UNIQUEIFY script calls CAD, UNIQUE and FREERFLAG for setting a FreeR_flag column. The latter may or may not be wanted, depending on whether the test-set has been assigned by XDS/XSCALE, already. Best regards, Dirk. Am 21.02.2008 um 16:15 schrieb [EMAIL PROTECTED]: In my experience when going from XDS via some intermediate file to mtz format, XDS uses in some cases a different reciprocal asymmetric unit as mtz uses, which may result in only half of the reflections being used and/or ccp4 programs getting confused. By using UNIQUE, one makes sure that the reflections are mapped to the correct asymmetric unit. It has nothing to do with missing reflections but is in many cases essential. Best regards, Herman -----Original Message----- From: CCP4 bulletin board [mailto:[EMAIL PROTECTED] On Behalf Of Kay Diederichs Sent: Thursday, February 21, 2008 3:46 PM To: CCP4BB@JISCMAIL.AC.UK Subject: Re: [ccp4bb] XDS and overlaps Simon Kolstoe schrieb: Whilst we are on the subject of XDS... I had difficulty processing a data-set in mosflm the other day so on the recommendation of a colleague switched to xds which, with a bit of tweaking, seemed to work really well. I converted the resulting XDS_ASCII.HKL using xdsconv and then f2mtz ready for phaser and refmac. We do it in the same way here. However, my colleague then told me that xds handled missing reflections differently from the usual mosflm/CCP4 route I have honestly not the slightest idea what your colleague was referring to. and thus I had to run the CCP4 program UNIQUE before I tried refinement as apparently refmac does not like reflection files originally processed with xds. As I couldn't In the case of a new project, one should run "uniqueify" or some other means of assigning reflections to the free set (thin shells come to mind; see earlier discussions on CCP4BB). In the case of an old project, one should transfer the free set of reflections from some master data set to the new dataset. None of this is specific to XDS. HTH, Kay find anything in the literature about this I was wondering whether this advice is still up to date? Thanks, Simon On 21 Feb 2008, at 09:44, Kay Diederichs wrote: Engin Ozkan schrieb: Hi everyone, I have been recently relying on XDS quite a bit, but at the same time worrying about how XDS treats overlaps. We had one dataset that both HKL2000 and Mosflm would show to have severe overlaps, as expected due to unit cell parameters and the unfortunate crystal orientation in the loop. We always ended up with completeness percentages in the 70's. XDS can find the same lattice, index and scale the data, but yields a 100% complete mtz (and a nice structure). Without the HKL/Mosflm-like GUI, it is difficult to assess the fate of the overlapped observations in XDS. What I could see with VIEW was that some observations were being divided into several ovals, probably different reflections, but I'm not very certain. So, the basic question is, how does XDS treat overlaps? I could not find in the documentation an answer to this question; the single mention of overlaps I could find tells me that XDS can recognize overlaps, but does not tell me if it rejects them, or divvies them up into separate reflections, and if that is the case, how does it divide them, and how reliable is that? Depending on how it divides the overlaps, could that affect commonly-used intensity stats and distributions? Thanks, Engin Engin, the basic answer is: a) each pixel of the detector is assigned to its nearest reflection in reciprocal space b) some of these pixels will mostly allow the background estimation, others will mostly contribute to the integration area (but as they are transformed into a local coordinate system there is not a 1:1 relationship). At this step, pixels which should be background but are higher than expected (due to overlap) are rejected. c) for each reflection, the background is estimated, and the 3D profile is assembled from the pixels contributing to it d) a comparison is made: for a reflection, is the percentage of its observed profile assembled in c) larger than some constant (called "MINPK" in XDS.INP)? If the answer is no, this reflection will be discarded (you could call this situation "overlap"). Among other things, this means that: a) the program does _not_ look around each reflection to detect an overlap situation, it just tries to gather the pixels for each reflection b) as a user, when your crystal-detector distance was chosen too low or the reflections are very broad (resulting in generally strong overlap), you may reduce MINPK down to 50. This will result in more completeness, but you should monitor the quality of the resulting data. Conversely, if you raise MINPK over its default of 75 you will discard more reflections, but the resulting dataset will be a bit cleaner. The reference is W. Kabsch (1988) Evaluation of single-crystal X-ray diffraction data from a position-sensitive detector. J. Appl. Cryst. 21, 916-924. (http://dx.doi.org/10.1107/S0021889888007903) HTH, Kay --Kay Diederichs http://strucbio.biologie.uni-konstanz.de email: [EMAIL PROTECTED] Tel +49 7531 88 4049 Fax 3183 Fachbereich Biologie, Universität Konstanz, Box M647, D-78457 Konstanz -- Kay Diederichs http://strucbio.biologie.uni-konstanz.de email: [EMAIL PROTECTED] Tel +49 7531 88 4049 Fax 3183 Fachbereich Biologie, Universität Konstanz, Box M647, D-78457 Konstanz ******************************************************* Dirk Kostrewa Gene Center, A 5.07 Ludwig-Maximilians-University Feodor-Lynen-Str. 25 81377 Munich Germany Phone: +49-89-2180-76845 Fax: +49-89-2180-76999 E-mail: [EMAIL PROTECTED] *******************************************************
