Since this discussion was actually started by the failure of SHELXE to print out RCullis and phasing power for reasons already explained (it does print out an "Estimated mean FOM" after density modification for what that is worth) I should point out that a very much better test of the quality of the map is how well it can be traced automatically. The criteria I use to judge this with the (beta-test) SHELXE that includes backbone tracing are the mean chain length (the longer the better, wrong maps give values under 10 residues) and the corrlation coefficient for the traced backbone against the native Fobs (values above 30% are invariably good solutions). I would always apply this test before investing more CPU time to complete the structure (e.g. with ARP/wARP). This is a simple and effective way of using our chemical knowledge to assess the quality of the phases.
George Prof. George M. Sheldrick FRS Dept. Structural Chemistry, University of Goettingen, Tammannstr. 4, D37077 Goettingen, Germany Tel. +49-551-39-3021 or -3068 Fax. +49-551-39-22582 On Wed, 14 Apr 2010, MARTYN SYMMONS wrote: > I agree this is very interesting - > > The 'real' FOM is a great idea (we should start a campaign for real FOMs!) > > What the 'real' FOM is trying to get at is the quality of the initial > experimental map. This could also be subsequently calculated for NCS-averaged > and density-modified maps subsequently to show how the building was achieved. > > Gamma correction and real space free residuals can be used to avoid > overfitting (DM has these I know, prob other progs too). But the point that > overfitting can help the crystallographer build a correct structure is a good > one. Conversely I have heard of structures that are not just correct but also > wonderfully original, built by talented crystallographers into initially > apparently difficult-to-interpret maps. So we maybe need to celebrate what > can be done; not use this as necessarily a criticism of the final structure. > > One of the things I heard from F. VonD was 'don't get hung up on statistics - > look at maps'. Any map is going to have ropey bits as well as easy bits - so > the only way to convey that would not be one number like a FOM. But more > like a residue-by-residue real-space Rfactor of the _final_ model in the > _original_ map? > > In the real world ;) I don't see that happening anytime soon.... > > all the best > Martyn > > Martyn Symmons > Cambridge > > > > > > ----- Original Message ---- > From: "Soisson, Stephen M" <stephen_sois...@merck.com> > To: CCP4BB@JISCMAIL.AC.UK > Sent: Tuesday, 13 April, 2010 19:09:12 > Subject: Re: [ccp4bb] Phasing statistics > > This is an interesting thread, and perhaps I should not dive in on such > a heady topic, BUT, I do want to point out my own particular bias > regarding FOM that is not entirely consistent with James' point of view. > In my experience, the FOM obtained after density modification runs are > almost always extremely optimistic - I have seen relatively high > apparent FOM after density modification runs (0.7) that had nearly > uninterpretable maps. As such, I am much more interested in knowing > about the FOM from the phasing calculations themselves and NOT after > density modification. > > That said, applying arbitrary cut-offs to what would be deemed an > acceptable FOM, after phasing calculations, to generate maps that are > "interpretable" is not really a good thing to do. For instance, I just > had a structure where the FOM was 0.35 after phasing (a rubbish > structure perhaps?), BUT, the data are highly redundant and the solvent > content in the high 70% range. The post density modified maps are > stunningly good. One could easily imagine many other scenarios (e.g. > NCS) where the modified maps and apparent FOM would be decoupled. > > So, I do agree with James's suggestion that perhaps we should be > retrospectively calculating a "real" FOM between the final model and the > actual maps you built into (after whatever you did to get them). This > seems like a very good idea indeed. > > More personal biases revealed: I actually look at, and use, the Cullis > R values on anomalous and isomorphous data to help determine how much > signal is in the data. Simplified, these numbers are your Average > estimated lack-of-closure divided by your average observed difference. > That's an important thing to know, and I find them quite useful. > > Steve > > -----Original Message----- > From: CCP4 bulletin board [mailto:ccp...@jiscmail.ac.uk] On Behalf Of > James Holton > Sent: Tuesday, April 13, 2010 1:48 PM > To: CCP4BB@JISCMAIL.AC.UK > Subject: Re: [ccp4bb] Phasing statistics > > Probably the only phasing stat that I pay any attention to these days is > > the Figure of Merit (FOM). This is because, the _definition_ of FOM is > that it is the cosine of the phase error (or at least your best estimate > > of it). FOM=1 is perfect phases and FOM=0 is random phases, and a > reasonable cutoff value for FOM is 0.5 (see Lunin & Woolfson, Acta D, > 1993). Yes, there are ways to get various programs to report very > inaccurate values for FOM (such as running DM for thousands of cycles), > and yes, there are often legitimate reasons to run these programs in > this way. But, there are also very wrong things one can do to get low > Rmerge, Rcryst, and especially Rfree. It is simply a matter of knowing > (and reporting) what you are doing. > > > If you are worried that your favorite estimate of FOM is inaccurate, > then you can always turn to your most accurate phases: those of your > final, refined model (the one that you have convinced yourself is > "right" and ready to publish). Taking these as the "true phases", the > "true FOM" can always be obtained by comparing the final-model phases to > > those of your initial map (using PHISTATS or SFTOOLS). This is by no > means standard practice, but perhaps it should be? > > > Anyway, FOM is _supposed_ to be the cosine of the phase error, and is > therefore the most relevant statistic when it comes to how good your > phases were when you started building. This is why it is important as a > > reviewer to know what it is. If I am faced with a structure that was > built into a MAD map with initial FOM = 0.8 to 2 A resolution, then I am > > already convinced that the structure is "right" because I know they had > a very clear map to build into. It is hard to do something egregiously > wrong with such a map (such as tracing it backwards), so I would even > excuse a high R/Rfree in this case, especially if the map has large > absent (disordered) regions that the authors were honest enough to not > build. > > > On the other hand, if the initial solvent-flattened SAD map had FOM=0.3 > to 2 A, you are really pushing it. It is possible to get a correct > structure from such a map, but extremely difficult. One might combine > some MR phases with the SAD phases to improve them somewhat, but how > does one evaluate such a result? I'd say that if FOM < 0.5, then the > phases don't make you right. You need to look at other statistics (like > > R/Rfree). > > > The extreme case, of course, is MR, where the "starting" FOM=0. The > author then makes an assumption about the starting phases (based on > prior knowledge such as homology with PDB ID = xxxx), and that > assumption is then borne out by an "acceptable" R/Rfree (Kleywegt & > Brunger, 1996). The "true FOM" (comparing final refined phases to those > > of the initial MR hit) in this case might still be interesting because > it tells you a lot about how much rebuilding had to be done. > > > To answer Frank's question about a 4 A structure with anisotropic > diffraction (which I assume means that 4 A is in the best direction, and > > the other(s) are 5 A or so), I would first ask that the "true" > resolution limit be denoted by the point where the average I/sigma drops > > to ~2 (this is _without_ an anisotropic resolution cutoff!). Then we > probably have a 4.5A structure. The "metrics by which we then judge the > > results?" then depends on the bigger question: "Does the evidence > presented justify the conclusions drawn?". If the conclusion is that > bond lengths in the active site are "strained", then the answer is > obviously "no". Indeed, if the conclusions rely on the helicies in a > 4.5 A map being traced in the right direction, then I would also answer > "no". This is because at 4.5 A the image of a backward-traced helix > looks a _lot_ like the correctly-traced one (see > http://bl831.als.lbl.gov/~jamesh/movies/index.html#reso). To put it > another way, the R-factors alone are not convincing evidence of a > correct trace at 4.5A, and corroborating evidence must be presented to > make the helix direction convincing. By "presented", I mean spelled out > > in the text, and by "corroborating evidence" I mean something as simple > as a clear connectivity with enough big side chains placed to deduce the > > register of the sequence. Barring that, something like "SeMet scanning" > > can also clarify tracing ambiguities (for a relevant example, see Chen > et al. (2007) PNAS 104 p 18999). I am not saying that every 4.5 A > structure needs to do this, but I am saying that the number of > alternative explanations (models) for a given observation (map) > increases as the map gets blurrier, and if a plausible alternative model > > could change the conclusions of the paper, then it must be eliminated > with controls. You know, basic science stuff. > > > It is a common misconception that MAD/SAD/MIR phasing depends on > resolution, but nowhere on the Harker diagram does one see the > "resolution" of the vectors. The accuracy of the phase depends entirely > > on the magnitude of the signal (delta-F) and the magnitude of the noise > (sigma(F)). This is why you only get experimental phases for strong > spots, and never all the way out to your "resolution limit". True, this > > is a "resolution dependence", but it is actually the signal-to-noise > ratio itself that is important. The only part of experimental phasing > that seems to be reproducibly resolution-dependent is the density > modification used to clean it up. This seems to be limited to pushing > your "good phases" out by ~ 1 A in most cases (i.e. from 4A to 3A or > from 3.5A to 2.5A, etc.), but I'm not sure why that is. Probably > something in histogram matching. Unfortunately, I am not aware of a > good comprehensive review of the resolution dependence of phase > extension, possibly because one cannot do such an analysis with the data > > currently available in the PDB (initial phases are not deposited). > > > I would finally like to note that I am highly uncomfortable with the > idea of excusing the reporting of data processing statistics if the > structure is deemed "correct". Formally, no protein structure is > intrinsically "correct" if it does not explain the data (Fobs) to > withing experimental error (~5%). In the "small molecule world" models > with Rcryst > Rmerge are rejected out-of-hand (and for good reason). > The only reason protein structures are "excused" from this rule is > because they have a good "track record" of agreeing with > experimentally-phased maps. > > -James Holton > MAD Scientist > > Frank von Delft wrote: > > I fully agree, for high quality data. > > > > What though if the data are not impeccable and the structure > > necessarily ropey? E.g. 4A phases and anisotropic diffraction. By > > what metrics do we then judge the results? > > > > (I don't know the answer, btw, but our membranous colleagues surely > > spend quite a bit of time with that question...) > > > > phx. > > > > > > On 12/04/2010 12:10, Anastassis Perrakis wrote: > >> Hi - > >> > >> A year or so ago, I have asked as a referee somebody to provide for a > > >> paper the statistics for their heavy atom derivative dataset, > >> and for the phasing statistics. For some good reasons, they were > >> unable to do that, and they (politely) asked me > >> 'what would it change if you knew these, isn't the structure we > >> present impeccable?'. Well, I think they were right. > >> Their structure was surely correct, surely high quality. After that > >> incident and giving it some thought, > >> I fail to see why should one report e.g. PP or Rcullis, or why will I > > >> care what they were if the structure has a convincing Rfree and is > >> properly validated. > >> If someone wants to cheat at the end of the day, its easy to provide > >> two numbers, but its hard to provide a good validated model that > >> agrees with the data. > >> (and, yes, you can also make up the data, but we have been there, > >> haven't we?!?) > >> > >> So, my question to that referee, likely being a ccp4bb aficionado > >> that is reading this email, or to anyone else really, is: > >> > >> "What would it help to judge the quality of the structure or the > >> paper if you know PP, Rcullis and FOM?" > >> > >> Best - > >> > >> A. > >> > >> PS Especially since you used SHELXE for phasing these statistics are > >> utterly irrelevant, and possibly you could advice the referee to read > > >> a bit about how SHELXE works ... or go to one of the nice courses > >> that George teaches ... > >> > >> On Apr 12, 2010, at 10:37, Eleanor Dodson wrote: > >> > >>> You can feed the SHELX sites into phaser_er or CRANK both of which > will > >>> give this sort of information. > >>> > >>> Or mlphare if you know how to set it up.. > >>> > >>> Eleanor > >>> > >>> > >>> Harmer, Nicholas wrote: > >>>> Dear CCP4ers, > >>>> > >>>> I've been asked by a referee to provide the phasing statistics for > >>>> a SAD dataset that I used to solve a recent structure. Whilst I > >>>> have been able to find a figure-of-merit for the data after > >>>> phasing, I can't work out how to get any other statistics (e.g. > >>>> phasing power or an equivalent or Rcullis). Does anyone know a good > > >>>> route to obtaining useful statistics to put in the paper for SAD > data? > >>>> > >>>> The structure solution was carried out using SHELX C/D/E and then > >>>> ARP/wARP. > >>>> > >>>> Thanks in advance, > >>>> > >>>> Nic Harmer > >>>> > >>>> ===================== > >>>> Dr. Nic Harmer > >>>> School of Biosciences > >>>> University of Exeter > >>>> tel: +44 1392 725179 > >>>> > >> > >> *P** **please don't print this e-mail unless you really need to* > >> Anastassis (Tassos) Perrakis, Principal Investigator / Staff Member > >> Department of Biochemistry (B8) > >> Netherlands Cancer Institute, > >> Dept. B8, 1066 CX Amsterdam, The Netherlands > >> Tel: +31 20 512 1951 Fax: +31 20 512 1954 Mobile / SMS: +31 6 28 > 597791 > >> > >> > >> > >> > Notice: This e-mail message, together with any attachments, contains > information of Merck & Co., Inc. (One Merck Drive, Whitehouse Station, New > Jersey, USA 08889), and/or its affiliates Direct contact information for > affiliates is available at http://www.merck.com/contact/contacts.html) that > may be confidential, proprietary copyrighted and/or legally privileged. It is > intended solely for the use of the individual or entity named on this > message. If you are not the intended recipient, and have received this > message in error, please notify us immediately by reply e-mail and then > delete it from your system. > >
