A way to achieve a feeling for the significance of the different map
features that you are looking at in your current project, is the old
control technique to omit both a well ordered part of the model
(residue, alpha-helical turn, part of a beta-strand, ...) and a less
well ordered part (residue, loop, end of visible secondary structure,
...). Then you run refinement until convergence and calculate the
different maps that you want to try. If you now look at the appearance
of these maps at different contour levels for both the well and less
well ordered omitted parts, and compare this with the appearance of the
same maps at the same contour levels for your questionable part (bound
ligand?), you can quickly get a feeling, how far you can trust what you
see in these maps.
I learned this at the beginning of my crystallographic work and still do
it today if I'm in doubt.
Cheers,
Dirk.
On 03.12.20 10:20, Robert Nicholls wrote:
Hi Dale,
You're absolutely right - the multiple hypothesis testing problem is
one that is often not considered, let alone properly accounted for.
Whilst this can be accounted for by appropriate adjustment of
significance levels when a known number of explicit hypotheses are
tested (and when estimated sigmas are appropriate and reliable...),
this is extremely difficult in the present context when we passively
conduct a large number of quick map evaluations subjectively by eye.
Objective guidelines in such a case, which don't essentially boil down
to an automated procedure, or unduly inhibit the process in other
ways, would be valuable. I don't think there's a clear answer to this
today, although raising awareness of such issues is very prudent.
Indeed, there is an outstanding need for additional approaches for
cross-validation, and perhaps re-evaluation of policies
regarding provision of evidence of the reproducibility of
crystallographic models. You're correct to say that, ultimately, there
is (presently) no substitute for education and experience.
Best regards,
Rob
On 3 Dec 2020, at 08:09, Dale Tronrud <de...@daletronrud.com
<mailto:de...@daletronrud.com>> wrote:
Hi,
Dr Nicholls brings up many interesting points, but doesn't touch on
the major point I had hoped to make in my letter. Whenever you start
making multiple tests of your hypothesis you have to evaluate each of
those tests with a higher standard than you would if you only applied
one. If you take a survey of the amount of fat people eat along with
their history of heart disease you can calculate a correlation and
find it significant with a p value of 0.05. If, instead, you perform
a survey asking for twenty different dietary behaviors and twenty
health outcomes and find a correlation between eating fat and heart
disease you need a much higher "signal" to determine its
significance. You just made 400 comparisons and a p of 0.05 allows
20 spurious correlations to appear significant.
If you are exploring your data set to decide if a compound has
bound, and your try several different refinement programs and
calculate several different map types based on the results of those
refinements, and then adjust the blur of each map, and pick the map
with the strongest peak in the putative binding site, you have to
consider the significance of that peak height to be less than if you
had just calculated one map and got that same height.
Ignoring this counterintuitive fact has resulted in a huge number
of studies in many fields to be published that ultimately turned out
to not be reproducible. It likely has also resulted in the
deposition of a lot of "complex" models in the PDB that aren't correct.
Yes, I am arguing for an ideal, hoping to pull some of you over
toward my side a bit. I certainly understand that one has to be
flexible when solving a difficult problem, but you can't ignore that
this "flexibility" has significant consequences for understanding the
results of your work.
Dr Nicholls' letter brings up a related topic which I'd like to
explore. His letter repeatedly mentions the importance of
"intuition" when interpreting a map. Yes, the power of human
intuition, and our inability to replicate it in silico is the reason
we are still staring at maps in Coot. Intuition is a remarkable tool
which, by its nature, is difficult to describe.
Yet, no one is born with an innate intuition for interpreting
electron density maps. Intuition is acquired thru practice.
Practice is not simple repetition, however. You can't become
proficient in shooting basketball hoops by simply repeatedly throwing
a basketball on the roof of your garage. You have to have a proper
backboard and a hoop. Now, after repeatedly throwing the ball and
"feeling" the difference between it going through the hoop and not,
you will develop the ability to make a basket w/o really thinking
about it. You will have developed an intuition for achieving that task.
There are two caveats. First, you have to actually watch the ball
go through the hoop. If you close your eyes right after your throw
you will never develop a useful skill. It is the feedback from the
success or failure of each attempt that makes it practice. Second,
no matter how much time you spend shooting baskets, you will never
get better at dribbling the ball. Good practice allows you to
develop intuition, but only intuition about that task.
Let's say you are working on a project, but having difficulty
interpreting your map at some critical location. You ask around and
learn of some spiffy new map calculation and you want to try it.
While you certainly can calculate the map, you have no intuition on
how to interpret it. You have not practiced with that type of map.
It may look similar to the maps you've looked at before, but that
similarity can be a trap. By now a large number of us here on the BB
have had the experience of looking at a high resolution electrostatic
potential (ESP) map and "feeling" that something is wrong with it.
The carbonyl oxygen bumps are too small and the acid groups are
oddly weak. Wow, those magnesium ions really stand out -- Maybe
they're potassium instead? No, there is nothing wrong with the ESP
map. The fault is with our intuition which was based on many, many
hours of looking at ED maps. To interpret ESP maps you have to
practice with a bunch of ESP maps first.
You cannot develop intuition for the spiffy map calculated from
your project's data since you don't know its correct interpretation
-- It cannot give you feedback. Before you calculate this map for
your data you should calculate versions for many other *completed*
projects and get a "feel" for what that kind of map shows under
different circumstances. Practice, practice, practice, then you will
be ready to return to your little mystery and be able to apply your,
newly acquired, intuition.
Yes, I try new refinement programs - But first I run refinement
with them on familiar proteins. Yes, I try new styles of map
calculations - But first I calculate those maps for cases where I
know the answer. I've refined a fair number of structures, probably
not as many as most of you, but at the end of a refinement I take the
answer and go back to the original maps. Looking at those maps in
light of the answer is what improves my map interpretation skills,
such as they are, the most.
All of my practice has been with ED (and some ESP) maps of better
than 3 A resolution. Despite all the intuition I can bring to bear
on them, when it comes to a 4 A resolution map I'm no better than an
undergrad.
Your first experience with a new technique should never be with
your current project's data. You should work to add that technique
to your tool box, and then move back to your data. Practice, and
more practice will build that squishy neural network in your head.
Descending from soapbox,
Dale Tronrud
On 12/1/2020 8:31 AM, Robert Nicholls wrote:
Dear all,
I feel the need to respond following last week’s critique of the use
of Coot’s map blurring tool for providing diagnostic insight and
aiding ligand identification…
On 24 Nov 2020, at 16:02, Dale Tronrud <de...@daletronrud.com
<mailto:de...@daletronrud.com> <mailto:de...@daletronrud.com
<mailto:de...@daletronrud.com>>> wrote:
To me, this sounds like a very dangerous way to use this tool
decide if a ligand has bound. I would be very reluctant to modify
my map with a range of arbitrary parameters until it looked like
what I wanted to see. The sharpening and blurring of this tool is
not guided or limited by theory or data.
I disagree with this, subject to the important qualification that
care is needed with interpretation. Blurring isn't a crime - it
merely involves adjusting the weighting given to lower versus higher
resolution reflections, and thus allows relaxation of the choice of
high-resolution limit, and facilitates local investigation of
regions that exhibit a poor signal-to-noise ratio. This is
particularly pertinent to liganded compounds, which are typically
present with sub-unitary occupancies.
Coot's blurring merely involves convolution of the whole map with an
isotropic 3D Gaussian, with a parameter (B-factor) to control the
standard deviation of the Gaussian. This corresponds to reweighting
the structure factors in order to give higher weight
to lower-resolution reflections. This approach is guided by a very
simple theory: higher resolution structure factors (SFs) are
typically noisier, with a worse signal-to-noise ratio than
lower resolution SFs (due to increased errors in both observed
higher-resolution reflections and calculated phases). Consequently,
increasing the blurring B-factor reduces the effect of the noisier
higher-resolution SFs. This results in a map that should be
more reliable, but at the expense of reduced structural detail due
to artificially reducing the effective resolution.
It should be noted that this does assume that lower resolution
reflections are more reliable than higher resolution ones. So, good
low-resolution data quality and completeness is important.
Unfortunately, determination of an optimal B-factor parameter is not
presently automated. Consequently, users are currently expected to
trial different values in the Coot slider tool in order to maximise
information and gain, for want of a better word, intuition.
Furthermore, due to the spatially heterogeneous nature of atomic
positional uncertainty in macromolecular complexes, it can be that
different B-factor parameters are of optimal usefulness in
different local regions of the map that exhibit
different signal-to-noise ratios. Such issues are on-going areas of
research.
The main problem is that interpretation is subjective. In difficult
cases, it is necessary to obtain as much information and insight as
possible in order to gain a good intuition. If you can't see a
ligand in the "standard" maps, but you can see evidence for a ligand
in blurred density (or difference density) maps of the
various types, then it means that careful exploration of those
avenues is required. Any "evidence" from viewing such maps and
map types should serve to guide intuition, and should be digested
along with all other available information. Such complementary maps
should be seen as diagnostics to gain intuition, rather than
something that can be used as an unequivocal argument for ligand
binding.
Ultimately, the presence of significant density in a blurred map
means that there is something substantial present. Or in a blurred
difference density that there is something missing from the current
model. This could be a missing ligand, or it could be a
mismodelled region of the macromolecule, or it could be mismodelled
solvent (in which case re-evaluating any solvent mask may be
worthwhile). Ultimately it is down to the practitioner to explore
all potential explanations for any such behaviour, in order to
maximise intuition and convince themselves of the crystal's
structural composition.
In some cases the presence of density in a blurred map might be
sufficient to convince the practitioner that it is worth pursing
investigation of binding. This may take various forms: hypothesising
an approximate pose for the ligand; the nature of interactions
in the structural environment of the macromolecule; re-evaluation
after modelling and refinement; or simply stating that there may be
evidence of binding. In many cases, the latter is the
appropriate action, and, as Robbie quite rightly pointed out: "in a
scientific setting this digging is not to come to a
strong conclusion, but only to see if you should pursue the project
and do additional experiments".
On 24 Nov 2020, at 16:02, Dale Tronrud <de...@daletronrud.com
<mailto:de...@daletronrud.com> <mailto:de...@daletronrud.com
<mailto:de...@daletronrud.com>>> wrote:
[...] to avoid bias in the interpretation of the results, all of
the statistical procedures are decided upon BEFORE the study is
even began. This protocol is written down and peer reviewed at the
start. Then the study is performed and the protocol is followed
exactly.
[...] I would recommend that you decide what sort of map you think
is the best at showing features of your active site, based on the
resolution of your data set and other qualities of your project,
before you calculate your first Fourier transform. If you think a
Polder map is the bee's knees then calculate a Polder map and live
with it. If you are convinced of the value of a FEM, or a Buster
map, or a SA omit map, or whatever, calculate that map instead and
live with it.
I agree that such an approach would be more scientific, and I
certainly find this idea very appealing. Whilst I hesitate to speak
against such a philosophy, I feel it is necessary to temper/balance
this view by pitching a counterargument in the interests
of pragmatism - in general it's just not that practical. And perhaps
propositions for revolution of best-practice policies within the
field should be distinct from current practical recommendation, in
the interests of avoiding potential confusion for the student/user
who simply wants a solution that they can apply to today's problems.
Whilst it sounds like a nice ideal, in general it is difficult to
know which pathologies might be encountered (e.g. ambiguous density
in the binding site; twinning; modelling difficulties around a
symmetry axis; multiple conformations;
semi-disorder; post-translational chemical modifications; radiation
damage… the list goes on). It's completely acceptable for
someone encountering a problem for the first time to explore what
tools are available to guide any decision-making, in the hope of
achieving the best model possible. A typical user cannot be expected
to outline a strategy for every eventuality a priori - that sounds
more like the design of an automated pipeline, not advice that users
should be expected follow.
In summary, it's unadvisable to put all eggs in one basket (of one
type of map, Polder or otherwise). If an experienced user likes a
particular tool because it's worked well for them in the past, it
doesn't mean that they shouldn't try other tools now (in this case:
view other types of maps) the next time they encounter a
problem. Especially given that tools in our field are still very
much evolving over time. Different approaches may have more value
and provide more insight in different circumstances.
Best regards,
Rob
------------------------------------------------------------------------
To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB&A=1
<https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB&A=1>
<https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB&A=1
<https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB&A=1>>
########################################################################
To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB&A=1
<https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB&A=1>
This message was issued to members of www.jiscmail.ac.uk/CCP4BB, a
mailing list hosted by www.jiscmail.ac.uk, terms & conditions are
available at https://www.jiscmail.ac.uk/policyandsecurity/
------------------------------------------------------------------------
To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB&A=1
<https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB&A=1>
--
*******************************************************
Dirk Kostrewa
Gene Center Munich
Department of Biochemistry, AG Hopfner
Ludwig-Maximilians-Universität München
Feodor-Lynen-Str. 25
D-81377 Munich
Germany
Phone: +49-89-2180-76845
Fax: +49-89-2180-76998
E-mail: dirk.kostr...@lmu.de
kostr...@genzentrum.lmu.de
WWW: www.genzentrum.lmu.de
*******************************************************
########################################################################
To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB&A=1
This message was issued to members of www.jiscmail.ac.uk/CCP4BB, a mailing list
hosted by www.jiscmail.ac.uk, terms & conditions are available at
https://www.jiscmail.ac.uk/policyandsecurity/
