Hi Frank and Ian, We struggled with the small changes in free R-factors when we implementing the paired refinement for resolution cut-offs in PDB_REDO. It's not just the lack of a proper test of significance for (weighted) R-factor changes, it's also a more philosophical problem. When should you reject a higher resolution cut-off? a) When it gives significantly higher R-factors (lenient) b) When it gives numerically higher R-factors (less lenient, but takes away the need for a significance test) c) When it does not give significantly lower R-factors (very strict; if I take X*sigma(R-free) as a cut-off, with X > 1.0, in most cases I should reject the higher cut-off).
PDB_REDO uses b), similar to Karplus and Diederichs in their Science paper. Then the next question is which metric are you going to use? R-free, weighted R-free, free log likelihood and CCfree are all written out by Refmac. At least the latter two have proper significance tests (likelihood ratios and transformation Z-scores respectively). Note that we use different models, constructed with different (but very much overlapping) data, but the metrics are calculated with the same data. The different metrics do not necessarily move in the same direction when moving to a higher resolution. We ended up using all 4 in PDB_REDO. By default a higher resolution cut-off is rejected if more than 1 metric gets (numerically) worse, but this can be changed by the user. Next question is the size of the resolution steps. How big should those be and how should they be set up? Karplus and Diederichs used equal steps in Angstrom, PDB_REDO uses equal steps in number of reflections. That way you add the same amount of data (but not usable information) with each step. Anyway, a different choice of steps will give a different final resolution cut-off. And the exact cut-off doesn't matter that much (see Evans and Murshudov). Different (versions of) refinement programs will probably also give somewhat different results. We tested our implementation on a number of structures in the PDB with data extending to higher resolution than marked in the PDB file and we observed that quite a lot had very conservative resolution cut-offs. In some cases we could use so much extra data that we could move to a more complex B-factor model and seriously improve R-factors. The best resolution cut-off is unclear and may change over time with improving methods. So whatever you choose, please deposit all the data that you can get even if you don't use it yourself. I think that the Karplus and Diederichs papers show us that you should at least realize that your resolution cut-off is a methodological choice that you should describe and should be able to defend if somebody asks you why you made that particular choice. Cheers, Robbie > On 1 September 2013 11:31, Frank von Delft <frank.vonde...@sgc.ox.ac.uk> > wrote: > > > > 2. > I'm struck by how small the improvements in R/Rfree are in > Diederichs & Karplus (ActaD 2013, > http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3689524/ > <http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3689524/> ); the authors > don't discuss it, but what's current thinking on how to estimate the expected > variation in R/Rfree - does the Tickle formalism (1998) still apply for ML with > very weak data? > > > > Frank, our paper is still relevant, unfortunately just not to the question > you're trying to answer! We were trying to answer 2 questions: 1) what > value of Rfree would you expect to get if the structure were free of > systematic error and only random errors were present, so that could be used > as a baseline (assuming a fixed cross-validation test set) to identify models > with gross (e.g. chain-tracing) errors; and 2) how much would you expect > Rfree to vary assuming a fixed starting model but with a different random > sampling of the test set (i.e. the "sampling standard deviation"). The latter is > relevant if say you want to compare the same structure (at the same > resolution obviously) done independently in 2 labs, since it tells you how big > the difference in Rfree for an arbitrary choice of test set needs to be before > you can claim that it's statistically significant. > > > In this case the questions are different because you're certainly not > comparing different models using the same test set, neither I suspect are > you comparing the same model with different randomly selected test sets. I > assume in this case that the test sets for different resolution cut-offs are > highly correlated, which I suspect makes it quite difficult to say what is a > significant difference in Rfree (I have not attempted to do the algebra!). > > > Rfree is one of a number of "model selection criteria" (see > http://en.wikipedia.org/wiki/Model_selection#Criteria_for_model_selectio > n) whose purpose is to provide a metric for comparison of different models > given specific data, i.e. as for the likelihood function they all take the form > f(model | data), so in all cases you're varying the model with fixed data. It's > use in the form f(data | model), i.e. where you're varying the data with a > fixed model I would say is somewhat questionable and certainly requires > careful analysis to determine whether the results are statistically significant. > Even assuming we can argue our way around the inappropriate application of > model selection methodology to a different problem, unfortunately Rfree is > far from an ideal criterion in this respect; a better one would surely be the > free log-likelihood as originally proposed by Gerard Bricogne. > > > Cheers > > > -- Ian >
