Dear Clemens,
Thank you so much! This was in the end the solution =) A lot of combining mtz files back and forth=) But as always one last thing remains... You write that I should make sure REFMAC reads in the test-set from the staraniso.mtz and does not replace it with a new one. This would only be the case if it generates a new test set when I initially import the reflection file, right? How can I check that retrospectively with my history-riddled file? cheers and thank you very much, Vrea PS: the links in [2] are not accurate anymore, the interface of CCP4i2 export alone already looks completely different and does not seem to want external processing files. >>> Clemens Vonrhein <[email protected]> 07/12/22 2:50 PM >>> Dear Vera, On Tue, Jul 12, 2022 at 02:04:51PM +0200, Vera Pfanzagl wrote: > >> What I initially did not know is that REFMAC and STARANISO do not go > >> well together when it comes to the Rfree-flags in the diffraction > >> cut-off regions. > > >Just to clarify: REFMAC should have no problems with STARANISO > >data. What you probably mean is that the unobservable reflections > >outside of the cut-off surface - which might be considered via an > >ellipsoidal aproximation - could be used for the DFc completion in > >REFMAC? > > > Yes this is what I meant, I just put it in very loose words =) but > for future reference, this is a good step to take, right? It might depend on how exactly you are running REFMAC (different GUIs/interfaces might use different flags and defaults). According to https://www2.mrc-lmb.cam.ac.uk/groups/murshudov/content/refmac/refmac_keywords.html#id.4d3ffe3d758a it says When calculating map coefficients REFMAC by default tries to restore missing reflections. Statistical basis of this is that expected value of unknown structure factors for missing reflections are better approximated using DFc than with 0 values. Which makes sense /if/ those missing reflections are expected to be larger than 0. This would be true for observations missed due to a cusp, overloads, detector module gaps etc. However, in the reciprocal space regions where we don't expect any observable intensity, these measurements are expected to be ~0 and should therefore not filled-in with D*Fc values. Think of a perfect, isotropically diffracting crystal where by any measure (CC1/2, I/sigI, Rpim etc) we would say that the highest diffraction limit is 3A: we would never compute map coefficients to 2A with everything in the 3-2A range consisting of D*Fc values just because we could generate Miller indices to any diffraction limit we chose ... right? The same is true for an anisotropic dataset - just that instead of an isotropic (sphere) cut-off surface (to e.g. 3A) we now have an anisotropic (approximated e.g. by an ellipsoid) cut-off surface (described by three numbers if approximated by an ellipsoid). We shouldn't compute/restore map coefficients in the reciprocal space regions where we don't expect any observable intensity at all. Of course, if the model is perfect, those Fc (Fcalc) values would be very close to zero anyway and the D attenuation factor also plays a role ... but since no model is perfect that can't always be guaranteed. > I will hopefully have a few more crystals with the same issue and I > would also now stick to this file. BUSTER understands the SA_flag generated by STARANISO, which defines those reflections that should get a D*Fc value (if no observation present) in sync with the anisotropic model it used. It will create three sets of 2mFo-DFc maps: only observations, filled-in within the anisotropic model and filled-in within the full diffraction limit sphere. REFMAC needs to be slightly fooled via the procedure described at [1] - and hopefully, the interface you use for driving REFMAC is not trying to be clever and generates again a complete set of test-set flags to the diffraction limit of the MTZ file. So make sure REFMAC sees the MTZ file exactly as prepared in [1]. > >Take the Data_1_autoPROC_STARANISO_all.cif and combine it [1] with the > >mmCIF from your refinement step: ensure that the data block with the > >reflection data that went /into/ your refinement is given as the first > >block. By convention, this is what e.g. PDB deposition/validation > >software will use for re-computing various statistics. > > >I would be inclined to go back to your original REFMAC refinement > >(before you did all that SFTOOLS, Phenix-again, server, conversion > >steps): do you have just an output MTZ file and no reflection mmCIF? > >Maybe by re-running the same job you could switch on some option to > >trigger writing of the reflection data in mmCIF format? Maybe even > >switching off the DFc completion and/or re-scaling? > > > PHENIX has this option to export the data as an mmcif but I cannot > find this in CPP4i2 (which is where I use REFMAC). Maybe the links on [2] are no longer accurate? > There is only the export and validation tool, but this is crashing > because it expects processing data which is not there and which i > cannot import either (I cannot select anything for Aimless XML). Maybe try the Gemmi [3] program instead (see 'gemmi convert -h' and 'gemmi mtz2cif -h')? > And here is where my > fluffy understanding of the mtz file format hits me over the head =) The MTZ format is rather simple here: it only holds reflection data and a few header items. It is the hunting for the data processing (merging) statistics that is the tricky part - which is why the CCP4i tool looks for an AIMLESS XML file it seems. > What I tried was to export the "old-style".mtz from CCP4i2 and then > to convert it using the gemmi server. But gemmi requires > intensities, which are missing in the REFMAC mtz, which contains > these arrays: H K L FREER F SIGF FC PHIC FC_ALL PHIC_ALL FWT PHWT > DELFWT PHDELWT FOM PHCOMB HLACOMB HLBCOMB HLCCOMB HLDCOMB FC_ALL_LS > PHIC_ALL_LS. Correct: output files from refinement programs often contain only the information relevant to the refinement job at hand (mainly: map coefficients) and drop a lot of other things. Can you maybe combine the input file to REFMAC with the output file - something like cad hklin1 input.mtz \ hklin2 refmac.mtz \ hklout input_refmac_combined.mtz \ <<e LABI FILE 1 ALL LABI FILE 2 E1=FWT E2=PHWT E3=DELFWT E4=PHDELWT E5=FC E6=PHIC E7=FOM e (sorry for the old-style sh-syntax) and then run gemmi mtz2cif input_refmac_combined.mtz input_refmac_combined.cif > Could it be that a large part of my problem is deived from the mini-mtz > output files that are generated by CPP4i2? There might be the need for a bit of MTZ file combination, yes ... > I think this is where I am stuck, I do not know how to get the > output mini-mtz together with the intensity data in one mtz file, or > where to convert this mtz to a cif (and also not how to combine this > with the cif file from autoproc, but this is a future problem) CAD is the perfect program for this (see above). So if you had three MTZ files: A = after data processing B = input to refinement C = output from refinement you could always do something like this: cad hklin1 A.mtz \ hklin2 B.mtz \ hklin3 C.mtz \ hklout ABC.mtz \ <<e LABI FILE 1 ALL LABI FILE 2 E1=FREE LABI FILE 3 E1=FWT E2=PHWT E3=DELFWT E4=PHDELWT E5=FC_ALL E6=PHIC_ALL E7=FOM LABO FILE 3 E1=FWT E2=PHWT E3=DELFWT E4=PHDELWT E5=FC E6=PHIC E7=FOM e (this assumes that all reflection data is indexed in the same way and have the same SG/setting and that the A.mtz contains both intensities and amlitudes ... your concrete situation might be slightly different) Cheers Clemens [1] https://staraniso.globalphasing.org/test_set_flags_about.html [2] https://www.wwpdb.org/deposition/PDBxDeposit [3] https://github.com/project-gemmi ######################################################################## To unsubscribe from the CCP4BB list, click the following link: https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB&A=1 This message was issued to members of www.jiscmail.ac.uk/CCP4BB, a mailing list hosted by www.jiscmail.ac.uk, terms & conditions are available at https://www.jiscmail.ac.uk/policyandsecurity/
