There was a comment on the bb from Ian Tickle last year which explains it very well, I think:
" " " Rfree is not unbiased: as a measure of the agreement it is biased upwards by overfitting (otherwise how could it be used to detect overfitting?), by failing to fit with the uncorrelated errors in the test-set Fobs, just as Rwork is biased downwards by fitting to the errors in the working-set Fobs. Overfitting becomes immediately apparent whenever you perform any refinement, so the only point at which there is no overfitting is for the initial model when Rwork and Rfree are equal, apart from a small difference arising from random sampling of the test-set (that sampling error could be reduced by performing refinements with all 20 working/test sets combinations and averaging the R values). From there on the 'gap' between Rwork and Rfree is a measure of the degree of overfitting, so we should really be taking some average of Rwork and Rfree as the true measure of agreement (though the biases are not exactly equal and opposite so it's not a simple arithmetic mean). The goal of choosing the appropriate refinement parameters, restraints and weights is to _minimise_ overfitting, not eliminate it. It is not possible to eliminate it completely: if it were then Rwork and Rfree would become equal (apart from that small effect from random sampling). " " " That was Ian, not me, of course! Best wishes, Jon Cooper. jon.b.coo...@protonmail.com -------- Original Message -------- On 20 Oct 2020, 04:27, Sam Tang wrote: > Hi, the question may be a bit weird, but how do you define 'over-fitting' in > the context of structure refinement? From users' perspective the practical > aspect is to 'fit' the model into the density. So there comes this question > from our juniors: fit is fit, how is a model over-fit? > > BRS > > Sam > > --------------------------------------------------------------- > > To unsubscribe from the CCP4BB list, click the following link: > https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB&A=1 ######################################################################## To unsubscribe from the CCP4BB list, click the following link: https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB&A=1 This message was issued to members of www.jiscmail.ac.uk/CCP4BB, a mailing list hosted by www.jiscmail.ac.uk, terms & conditions are available at https://www.jiscmail.ac.uk/policyandsecurity/
