Dear Ed,
my head starts smoking ;-)
While it's clear that the Rfree goes down when a structure becomes
better during refinement, I think, its not the correlation of the
_changes_ of |Fobs-Fcalc|, but correlation of the _final_ |Fobs-
Fcalc| that is important here. I don't know the answer right now, but
maybe the following short summaries help to clarify the topic a bit
further (also for myself).
If we define bias as an unwanted correlation between Rwork and Rfree
thereby artificially lowering the Rfree too close to Rwork, we're
left with analyzing any correlation of the final |Fobs-Fcalc| between
working set and test set.
Let's have a closer to look to the two different main cases (letting
aside correlation between different data sets). I first look at
correlation between Fobs, then between Fcalcs and finally between |
Fobs-Fcalc|.
(1) In the general case of NCS, excluding pseudo-higher symmetry
cases, we will find only by chance Fobs in the test set that
correlate with Fobs' in the working set if they come very close to
each other after application of the NCS-operation. The overall
correlation in Fobs will be rather low (except in cases of very high
NCS, maybe).
(1a) If we apply NCS-constraints/restraints during refinement, we
exploit any NCS-correlation in the observed data and thereby
introduce both correlation between NCS-related Fcalcs and correlation
to the corresponding NCS-related working set Fobs. As a result, we
introduce some correlation between the final |Fobs-Fcalc| in the
working set and the test set, i.e. we introduce bias. The amount of
bias will depend on the choice of the test set, the degree of NCS
symmetry and how similar the NCS-copies are forced to become during
refinement.
(1b) If we do not apply any NCS-constraints/restraints during
refinement, the amount of correlation between NCS-related Fcalcs
depends on how similar the NCS-copies independently refine and how
close the corresponding NCS-related Fobs come in the working set and
the test set. Without application of NCS-constraints/restraints, I
would expect that the correlation between final |Fobs-Fcalc| in the
working set and the test set will be low.
(2) In the special case of pseudo-higher symmetry NCS or
intentionally treating higher symmetry data in a lower symmetry, we
will find Fobs in the test set that are highly correlated to higher
symmetry-related Fobs' in the working set.
(2a) If we apply the NCS-symmetry during refinement or even refine in
the higher symmetry, we introduce both correlation between NCS-/
higher symmetry-related Fcalcs and correlation to the corresponding
NCS-/higher symmetry-related working set Fobs. Thus, the final |Fobs-
Fcalc| in the working set and the test set will be highly correlated
and there will be strong bias between Rfree and Rwork.
(2b) If we do not apply NCS-symmetry or higher symmetry during
refinement, the amount of correlation between NCS-/higher symmetry-
related Fcalcs depends on the degree of similarity of the
independently refined copies. If they are still similar, there will
be both correlation between the NCS-/higher symmetry-related Fcalcs
and correlation to the working set Fobs. Consequently, there will be
still correlation of the final |Fobs-Fcalc| between working set and
test set, i.e. bias of Rfree. The amount of bias depends only on how
similar the independently refined copies become. In case of true
higher symmetry, I expect them to be very similar and thus, the
resulting bias will be rather high.
How far can we agree on this?
I would also like to hear the opinion of other crystallographers that
are mathematically more trained than me, or did we loose attention on
this topic, already?
Best regards,
Dirk.
Am 11.02.2008 um 22:50 schrieb Edward Berry:
Dirk Kostrewa wrote:
Dear Ed,
although, I don't think that a comparison of refinement in a
higher and a lower symmetry space group is valid for general NCS
cases, I will try to answer your question. Here are my thoughts
for two different cases:
(1) You have data to atomic resolution with high I/sigma and low
Rsym (I assume high redundancy). The n copies of the asymmetric
unit in the unit cell are really identical and obey the higher
symmetry (so, not a protein crystal). When you process the data in
lower symmetry (say, P1), the non-averaged "higher-symmetry"-
equivalent Fobs will differ due to measurement errors, and thus
reflections in the working-set will differ to "higher-symmetry"-
related reflections in the test-set due to these measurement
errors. If you then refine the n copies against the working-set in
the lower P1 symmetry, you minimize |Fobs(work)-Fcalc|, resulting
in Fcalcs that become closer to the working-set Fobs. As a
consequence, the Fcalcs will thus diverge somewhat from the test-
set Fobs. However, since this atomic model is assumed to be very
well defined obeying the higher symmetry, and, furthermore, the
working-set contains well measured "higher-symmetry"-equivalent
Fobs, the resulting atomic positions, and thus the Fcalcs, will be
very close to their equivalent values in the higher-symmetry
refinement. Therefore, the Fcalcs will also be still very similar
to the "higher-symmetry"-equivalent Fobs in the test-set, and I
would expect a difference between Rwork and Rfree ranging from "0"
to the value of Rsym. In other words, the Fobs in the test-set are
not really independent of the reflections in the working-set, and
thus Rfree is heavily biased towards Rwork.
In this case, I would not expect large differences in the outcome
due to the additional application of "NCS"-constraints/restraints.
As I see it, this is clearly a case of |Fo-Fc| for the test reflectins
decreasing because the model is getting better, and there is no bias.
Lets say the higher symmetry really does apply, so the correct
structure
is perfectly symmetrical and the "NCS-related" reflections agree to
within
the error level.
Lets also say the initial model is perfectly symmetrical (you
solved the
molecular replacement with two copies of the same monomer, and rigid-
body refinement positioned them exactly). But let's say it is
completely
unrefined- the search model is from a different organism in a
different
space group, and modified by homology modeling to your sequence.
So the Fo obey the NCS within error, The Fc obey the NCS, but the
Fobs don't fit the Fcalc very well. Initially there is no Free-R bias,
because the model has not been refined agaist the data. The free set
can only be biased by refinement, since it is only during refinement
that the the free set is treated differently. Thus it doesn't matter
that the ncs-related Fo are correlated and the ncs-related Fc
are correlated: it is only the CHANGES in Fc that could introduce
model bias, and they are uncorrelated if you do not enforce ncs.
Now as we refine, the model will converge toward the correct
symmetrical
model as a result of minimizing the |Fo-Fc| for the work reflections.
At the same time the |Fo-Fc| for the test reflections will also
decrease
on the average, but to a lesser extent. I argue that the only
mechanism
for refinement to reduce |Fo-Fc| at a test reflection is by improving
the structure, and I think that constitutes an unbiased Free-R value.
If you can think of any mechanism to reduce |Fo-Fc| for a test
reflection
because you are refining against a symm-related work reflection, then
the R-free would be biased. This is not the case if you do not
enforce
symmetry. On the average no decrease in |Fo-Fc|(test) will result from
changes that reduce |Fo-Fc| for the work reflection: given an
arbitrary
change in the structure, the change in |Fc| at arbitrary reflections
is a pseudo-random variable with expected value zero, and there is no
correlation between the change at ncs-related reflections.
The value of |Fo-Fc| at a test reflection goes down, not due to
changes which improve the fit at a sym-related working reflection,
but because of changes that improve the fit at all test reflections,
and then only because the structure is improving. The atoms moved into
symmetrical positions not because they were constrained to do so,
but because that fits the data better, in turn because the true
structure
is symmetrical. If the symmetry doesn't hold for some atoms, they will
tend to move into asymmetric positions to minimize |Fo-Fc| at work
reflections, now *decreasing* the correlation with sym-related work
reflections. But again this will tend to reduce |Fo-Fc| at free
reflections, simply because the model is better approximating the
true structure.
To make a more obvious parallel, suppose you are refining a low-
resolution
dataset from a microcrystal (with no NCS). In another directory on the
same disk you have a high resolution structure refined against a
larger
but isomorphous crystal from the same well, same cryo treatment,
using a different or no free set. The Fo's will be highly correlated
between the two dataets, because they are isomorphous crystals
of the same protein.
Now if you constrain your low resolution model to be close to
the high resolution one, your free set will be biased because
those reflections were used in refining the other structure,
and you are constraining the new structure to be the same.
If you DON'T impose any restraints between the two models, the
new model will STILL tend toward the high-resolution structure,
because it is a good approximation of the true structure.
Hence the Fc's will become highly correlated to the Fc's of
that structure. And |Fo-Fc| of the test reflections will decrease,
not because the structural changes you are making improved the fit
of the high-resolution structure to the reflection in that dataset
which is a test reflection in the new dataset, but only because
the model is improving.
Using your logic, because the model (and hence Fc's) are approaching
those of the structure which was refined against the test reflections,
so the test reflections must be biased.
Thanks for taking the time to help me work this out,
Ed
(2) You have data to non-atomic lower resolution, weak I/sigma and
poor Rsym. It is impossible to say whether the n copies of the
asymmetric unit in the unit cell are really identical, but they
are treated so assuming the higher symmetry (so, a real protein
crystal). For data processing, the same holds true as for case
(1). In contrast, here I think that it makes a difference, whether
you apply "NCS"-constraints/restraints between the n copies in the
lower symmetry P1, or not. If you apply "NCS"-constraints or
strong "NCS"-restraints, the n copies are made equal and you get n
times the average structure. This is similar to the refinement in
the higher symmetry, except that again you minimize the
discrepancy between Fcalcs and working-set Fobs, which will
increase the discrepancy to the "higher-symmetry"-related Fobs in
the test-set. But since the Fobs in the test-set are still not
really independent to the Fobs in the working-set, I would again
expect maximum differences between Rwork and Rfree in the same
order of magnitude as Rsym. So, Rfree is still biased towards
Rwork, but it might be more difficult to notice this. But if you
do not apply "NCS"-constraints/restraints, you give the less well-
defined atomic model more freedom to converge against the working-
set Fobs, resulting in a higher discrepancy between Rwork and
Rfree. But since the Fobs in the working set still contain "higher-
symmetry"-equivalent Fobs, you will end up with a model that still
shows some similarity to the refined structure in the higher
symmetry. As a result, the Rfree is even then not really
independent of Rwork, but it might be even more difficult to
notice this, depending on data resolution and quality. Here, I
can't give a range of differences between Rwork and Rfree.
So, this is still not quantitative, and I hope that I'm not
completely wrong with my argumentation.
These lower vs. higher symmetry examples given above are only
transferable to reality in special NCS-cases with pseudo-higher
symmetry (what Dale Tronrud discussed). Taking these special cases
aside, what do the NCS experts say to my original statement that
precautions against NCS bias in Rfree must only be taken if NCS-
constraints/restraints are really applied during refinement?
Best regards,
Dirk.
*******************************************************
Dirk Kostrewa
Gene Center, A 5.07
Ludwig-Maximilians-University
Feodor-Lynen-Str. 25
81377 Munich
Germany
Phone: +49-89-2180-76845
Fax: +49-89-2180-76999
E-mail: [EMAIL PROTECTED]
*******************************************************
