Hi Ian,
consider the case where two data sets have been collected from the
same crystal (or a crystal from the same drop), each processed
separately, and the structure refined against one of the two data sets
until convergence. The two data sets will be somewhat different due to
measurement errors but still very similar. Thus, when I take the
refined structure and re-refine it against the second data set using
the same indices for working and test set (and the same refinement
parameters), both the starting R and Rfree will not have converged
against the second data set, but will be similar to the refined values
from the first data set. The differences will be mainly caused by the
measurement errors. It is this type of bias of the test set that (at
least) I mean. After convergence of refinement against the second data
set, both R and Rfree will be then very similar for the two data sets.
Best regards,
Dirk.
Am 24.09.2009 um 11:56 schrieb Ian Tickle:
Hi, I beg to disagree with the 'perceived wisdom', including just
about
everyone on this BB, but my answer is NO, there should be no bias -
*provided* you do the subsequent refinement properly. First off,
Rfree
is useless as any kind of statistical measure of overfitting etc
*unless* the refinement has converged to the point of maximum log
likelihood against the current working set. So it's meaningless to
say
that Rfree is biased 'initially' i.e. *before* any further
refinement is
done using the new data because Rfree with the new data has no meaning
at that point - it's neither biased nor unbiased, it's just
meaningless!
In any case why would one want to report an Rfree *before*
refinement -
what use is it?
So we can only sensibly talk about the Rfree values *after* the
further
refinement has converged - and if the refinement hasn't converged then
Rfree bias is the least of your worries! So are people really saying
that the Rfree at convergence using the new data is biased? For
that to
be true it would have to be possible to arrive at a different unbiased
Rfree from another starting point. But provided your starting point
wasn't a local maximum LL and you haven't gotten into a local maximum
along the way, convergence will be to a unique global maximum of the
LL,
so the Rfree must be the same whatever starting point is used (within
the radius of convergence of course).
The other cures suggested such as SA and randomisation are IMO at
best a
waste of time and effort (i.e. it will take longer for subsequent
refinement to recover from the shock to the system), and at worst
likely
to be worse than the disease they purport to cure. For example how do
you know what RMS shift to use in the randomisation without causing
the
structure to jump into a local maximum LL: the resulting Rfree will
certainly be biased then!
There is of course a different issue (and maybe this is what is
confusing some people) of comparing Rfree's from different test
sets: we
showed that this introduces a random relative error in Rfree of
1/sqrt(2*Nfree) (where Nfree = size of test set). However this effect
is not bias, it's random sampling error.
Cheers
-- Ian
-----Original Message-----
From: owner-ccp...@jiscmail.ac.uk [mailto:owner-
ccp...@jiscmail.ac.uk]
On
Behalf Of Mike England
Sent: 24 September 2009 04:31
To: CCP4BB@JISCMAIL.AC.UK
Subject: Rfree in similar data set
Hi all,
I will appreciate your comments on the following case:
I have two datasets from the same or identical crystals. Initially, I
refine a structure against the first data set and later on switch to
another dataset for further refinements.
Do you think, my Rfree will be biased as Rfree reflections in second
dataset may be in fact Rwork reflections in previous datasets ?
Thanks in advance,
Mike
Disclaimer
This communication is confidential and may contain privileged
information intended solely for the named addressee(s). It may not
be used or disclosed except for the purpose for which it has been
sent. If you are not the intended recipient you must not review,
use, disclose, copy, distribute or take any action in reliance upon
it. If you have received this communication in error, please notify
Astex Therapeutics Ltd by emailing i.tic...@astex-therapeutics.com
and destroy all copies of the message and any attached documents.
Astex Therapeutics Ltd monitors, controls and protects all its
messaging traffic in compliance with its corporate email policy. The
Company accepts no liability or responsibility for any onward
transmission or use of emails and attachments having left the Astex
Therapeutics domain. Unless expressly stated, opinions in this
message are those of the individual sender and not of Astex
Therapeutics Ltd. The recipient should check this email and any
attachments for the presence of computer viruses. Astex Therapeutics
Ltd accepts no liability for damage caused by any virus transmitted
by this email. E-mail is susceptible to data corruption,
interception, unauthorized amendment, and tampering, Astex
Therapeutics Ltd only send and receive e-mails on the basis that the
Company is not liable for any such alteration or any consequences
thereof.
Astex Therapeutics Ltd., Registered in England at 436 Cambridge
Science Park, Cambridge CB4 0QA under number 3751674
*******************************************************
Dirk Kostrewa
Gene Center, A 5.07
Ludwig-Maximilians-University
Feodor-Lynen-Str. 25
81377 Munich
Germany
Phone: +49-89-2180-76845
Fax: +49-89-2180-76999
E-mail: kostr...@genzentrum.lmu.de
WWW: www.genzentrum.lmu.de
*******************************************************
