For those who may not have made it through all the CCP4bb postings in
October-December 2011 on archiving raw images, I have posted a summary at the
IUCR Diffraction Data Deposition Working Group forum page
http://forums.iucr.org/viewforum.php?f=21 in which I have attempted to list
the unique
A few responses regarding TARDIS:
While it's true that TARDIS.edu.au is just an index to files hosted elsewhere,
a new solution, MyTARDIS has been developed as a repository that holds
information on diffraction (and other kinds of) data. The idea being that
facilities, labs or institutions take
On Thu, 2011-10-27 at 15:36 +0200, George M. Sheldrick wrote:
> In non-continuous mode, the goniometer has
> to accelerate at the start of a frame and decellerate at the end, then
> wait for
> the frame to be read.
Someone should be able to confirm this, but I was under impression that
at synchrot
ic.es
>
>
> -Mensaje original-
> De: CCP4 bulletin board [mailto:CCP4BB@JISCMAIL.AC.UK] En nombre de Colin
> Nave
> Enviado el: jueves, 27 de octubre de 2011 0:49
> Para: CCP4BB@JISCMAIL.AC.UK
> Asunto: Re: [ccp4bb] IUCr committees, deposi
mation
>>
>> Where exactly is this existing store to which you refer?
>>
>>
>> The PDB is out it seems to take on this role for the MX community so it
>> leaves us with convincing the Journals. I have started with IUCr Journals
>> and have submitted a Business Plan, already a month ago.
>> It has not instantly led to 'lets do it&
This discussion of image deposition and archival has certainly been
illuminating. While there have been clear directions on how the process would
actually work, I am becoming increasingly curious on why it should be done
(outside of threatening non publication, social acceptance, funding, etc).
tíficas
Spanish National Research Council
www.csic.es
<http://www.csic.es/> Sin título-1
De: David Waterman [mailto:dgwater...@gmail.com]
Enviado el: jueves, 27 de octubre de 2011 12:05
Para: Martin M. Ripoll
CC: CCP4BB@jiscmail.ac.uk
Asunto: Re: [ccp4bb] IUCr committees, depositing i
sn't mean 'No'.
>
> I copy in Tom and Brian.
>
> In anticipation of your clarification.
>
> Thankyou.
> Yours sincerely,
> John
> Prof John R Helliwell DSc
===========
(Below, the mess
rames have sufficiently fine intervals (given by
> some sampling theorem consideration) I can't see how one loses information.
> Can one of you explain?
> Thanks
> Colin
>
>
>
> -Original Message-----
> From: CCP4 bulletin board [mailto:CCP4BB@JISCMAIL.AC.UK] On B
Hi James,
1) thanks for sending and email in this thread longer than mine, I was
"worried" I had killed it... ;)
2) you say:
> Of course, if we are willing to relax the requirement of validation and
> curation, this could be a whole lot easier. In fact, there is already
> an image deposition i
inal-
De: CCP4 bulletin board [mailto:CCP4BB@JISCMAIL.AC.UK] En nombre de Colin
Nave
Enviado el: jueves, 27 de octubre de 2011 0:49
Para: CCP4BB@JISCMAIL.AC.UK
Asunto: Re: [ccp4bb] IUCr committees, depositing images
Dear George, Martin
I don't understand the point that one is thro
On Wednesday, 26 October 2011, James Holton wrote:
> Of course, if we are willing to relax the requirement of validation and
> curation, this could be a whole lot easier. In fact, there is already
> an image deposition infrastructure in place! It is called TARDIS:
>
> http://tardis.edu.au/
>
ac.uk/
Diamond Light Source, Chilton, Didcot, Oxon, OX11 0DE, U.K.
From: CCP4 bulletin board [mailto:CCP4BB@JISCMAIL.AC.UK] On Behalf Of Matthew
BOWLER
Sent: 26 October 2011 10:03
To: ccp4bb
Subject: Re: [ccp4bb] IUCr committees, depositing images
The archiving of all raw data and subsequently m
ginal Message-
From: James Holton [mailto:jmhol...@lbl.gov]
Sent: 27 October 2011 06:04
To: Nave, Colin (DLSLtd,RAL,DIA)
Cc: ccp4bb
Subject: Re: [ccp4bb] IUCr committees, depositing images
In the spirit of supporting a "can do" attitude, I have decided to try
and frame the binar
the PDB the same release rules should apply for the
images as for the other data. For data not (yet) in the PDB, the funders of the research
might want to define release rules. However, we can make suggestions!"
The original had "For other data" rather than "For data not (yet) i
in the PDB"
-Original Message-
From: Gerard Bricogne [mailto:g...@globalphasing.com]
Sent: 26 October 2011 23:23
To: Nave, Colin (DLSLtd,RAL,DIA)
Cc: ccp4bb
Subject: Re: [ccp4bb] IUCr committees, depositing images
Dear Colin,
Thank you for accepting the heavy burden of respons
anks
Colin
-Original Message-
From: CCP4 bulletin board [mailto:CCP4BB@JISCMAIL.AC.UK] On Behalf Of Martin M.
Ripoll
Sent: 26 October 2011 22:50
To: ccp4bb
Subject: Re: [ccp4bb] IUCr committees, depositing images
Dear George, dear all,
I was just trying to summarize my point of view regarding this
es (one
> from each lattice) each with less disorder rather than just one average
> structure. Not sure whether Gloria's modulated structures would be as
> ubiquitous but her argument is along the same lines.
>
> Regards
> Colin
>
> -Original Message-
> Fr
o el: miércoles, 26 de octubre de 2011 11:52
Para: CCP4BB@JISCMAIL.AC.UK
Asunto: Re: [ccp4bb] IUCr committees, depositing images
This raises an important point. The new continuous readout detectors such as
the
Pilatus for beamlines or the Bruker Photon for in-house use enable the
crystal to
be rot
e average structure.
Not sure whether Gloria's modulated structures would be as ubiquitous but her
argument is along the same lines.
Regards
Colin
-Original Message-
From: CCP4 bulletin board [mailto:CCP4BB@JISCMAIL.AC.UK] On Behalf Of Herbert
J. Bernstein
Sent: 26 October 2011
On Oct 26, 2011, at 9:59 AM, Patrick Shaw Stewart wrote:
> The principle is that the movie is written to the same area of memory,
> jumping back to the beginning when it is full (this part is not essential,
> but it makes the principle clear). Then, when the photographer takes his
> finger of
Dear Colleagues,
Gerard strikes a very useful note in pleading for a "can-do"
approach. Part of going from "can-do" to "actually-done"
is to make realistic estimates of the costs of "doing" and
then to adjust plans appropriately to do what can be afforded
now and to work towards doing as much
Touche! But alas, I have no access to the PDB's server, so...
JPK
On Wed, Oct 26, 2011 at 11:54 AM, Frank von Delft
wrote:
> Cool - we've found our volunteer!!
>
> On 26/10/2011 17:28, Jacob Keller wrote:
>>
>> Is anyone seriously questioning whether we should archive the images
>> used for publ
Cool - we've found our volunteer!!
On 26/10/2011 17:28, Jacob Keller wrote:
Is anyone seriously questioning whether we should archive the images
used for published structures? That amount of space is trivial, could
be implemented just as another link in the PDB website, and would be
really helpf
Is anyone seriously questioning whether we should archive the images
used for published structures? That amount of space is trivial, could
be implemented just as another link in the PDB website, and would be
really helpful in some cases. One person could set it up in a day! You
could just make it a
I just want to jump in to state that I am ALL FOR the notion of
depositing the images that go with the structure factors and the
refined structure.
Through the years, I have been interviewing folks about the strange
satellite diffraction they saw, but ignored,
used the mains that they could integr
Could you perhaps use the principle of "capture storage" that is used by
wild-life photographers with high-speed cameras?
The principle is that the movie is written to the same area of memory,
jumping back to the beginning when it is full (this part is not essential,
but it makes the principle cle
Dear John and colleagues,
There seem to be a set a centrifugal forces at play within this thread
that are distracting us from a sensible path of concrete action by throwing
decoys in every conceivable direction, e.g.
* "Pilatus detectors spew out such a volume of data that we can't
poss
Dear Frank,
re 'who will write the grant?'.
This is not as easy as it sounds, would that it were!
There are two possible business plans:-
Option 1. Specifically for MX is the PDB as the first and foremost
candidate to seek such additional funds for full diffraction data
deposition for each future
Dear James,
Good analysis! You bring up important points.
On 10/24/11 23:56, James Holton wrote:
The Pilatus is fast, but or decades now we have had detectors that can read
out in ~1s. This means that you can collect a typical ~100 image dataset in a
few minutes (if flux is not limiting). Since
This raises an important point. The new continuous readout detectors such as the
Pilatus for beamlines or the Bruker Photon for in-house use enable the crystal
to
be rotated at constant velocity, eliminating the mechanical errors associated
with
'stop and go' data collection. Storing their data
The archiving of all raw data and subsequently making it public is
something that the large facilities are currently debating whether to
do. Here at the ESRF we store user data for only 6 months (and I
believe that it is available longer on tape) and we already have trouble
with capacity. My
Hi James,
Just to pick up on your point about the Pilatus detectors. Yesterday
in 2 hours of giving a beamline a workout (admittedly with Thaumatin)
we acquired 400 + GB of data*. Now I appreciate that this is not
really routine operation, but it does raise an interesting point - if
you have loade
Since when has the cost of any project been limited by the cost of
hardware? Someone has to /implement /this -- and make a career out of
it; thunderingly absent from this thread has been the chorus of
volunteers who will write the grant.
phx
On 25/10/2011 21:10, Herbert J. Bernstein wrote:
On Tue, Oct 25, 2011 at 3:20 PM, Pete Meyer wrote:
> This is probably an idea that has already been tried (or discarded as
> unsuitable for reasons that don't occur to me at the moment) - but why not
> start with good crystals (such as lysozyme) and deliberately make them
> worse? Exactly how wo
James Holton wrote:
I go to a lot of methods meetings, and it pains me to see the most
brilliant minds in the field starved for "interesting" data sets. The
problem is that it is very easy to get people to send you data that is
so bad that it can't be solved by any software imaginable (I've go
To be fair to those concerned about cost, a more conservative estimate
from the NSF RDLM workshop last summer in Princeton is $1,000 to $3,000
per terabyte per year for long term storage allowing for overhead in
moderate-sized institutions such as the PDB. Larger entities, such
as Google are able
On Oct 24, 2011, at 3:56 PM, James Holton wrote:
> The PDB only gets about 8000 depositions per year
Just to put this into dollars. If each dataset is about 17 GB in size, then
that's about 14 TB of storage that needs to come online every year to store the
raw data for every structure. A two s
Dear James,
This is technically ingenious stuff.
Perhaps it could be applied to help the 'full archive challenge' ie containing
many data sets that will never lead to publication/ database deposition?
However for the latter,publication/deposition, subset you would surely not
'tamper' with the r
The Pilatus is fast, but or decades now we have had detectors that can
read out in ~1s. This means that you can collect a typical ~100 image
dataset in a few minutes (if flux is not limiting). Since there are
~150 beamlines currently operating around the world and they are open
about 200 days
John Helliwell points out to me that it might be useful to know what MX
crystallographic data researchers in different countries are already
expected to deposit or save. He notes that research funding agencies in
the UK expect researchers to preserve their raw experimental data for at
least 5 years
Hi Eleanor,
So far I have managed to "lurk" on this one - keeping an eye on things
but not getting involved. However this has prompted me to respond!
> Has anyone raided the point that while archiving is good, it will only be
> generally useful if the image HEADERS are informative and use a
> co
Has anyone raided the point that while archiving is good, it will only
be generally useful if the image HEADERS are informative and use a
comprehensible format - and the data base is documented...
Eleanor
On 10/19/2011 10:45 AM, Alun Ashton wrote:
‘Short’ bit:
Of Tom Peat
Sent: 18 October 2011 23:29
To: ccp4bb
Subject: Re: [ccp4bb] IUCr committees, depositing images
If we are talking schemes, here is another one that we use that might be
considered:
Date/person/project/barcode/well#/crystal#
At the Australian synchrotron, a directory is automatically
123
> http://www.rsc.org/shop/books/2008/9780854042722.asp
>
>
> Original message
>> Date: Tue, 18 Oct 2011 18:17:14 +0100
>> From: CCP4 bulletin board (on behalf of Gerard
>> Bricogne )
>> Subject: Re: [ccp4bb] IUCr committees, depositing images
>>
What do you consider good ? r.m.s.d of 2.5 Å ? fatal for drug design.
On Oct 18, 2011, at 12:19 PM, Enrico Stura wrote:
Dear Peter,
How many crystallographers does it take to transform bad data into good
data?
None, you need a modeller. Only a modeller can give you a structure with
perfect
geome
, Oct 18, 2011 12:01 pm
Subject: Re: [ccp4bb] IUCr committees, depositing images
One more consideration:
Since organization is not one of my greatest talents, I would be absolutely
delighted if a databank took over the burden of archiving my raw data for me.
Phoebe
==
On Tue, 2011-10-18 at 18:17 +0100, Gerard Bricogne wrote:
> it would be of the greatest value to that
> investigator to be able to double-check how reliable some features of
> that
> structure (especially its ligands) actually are.
Certainly, one could argue here that the current PDB policy that
behalf of Gerard
>Bricogne )
>Subject: Re: [ccp4bb] IUCr committees, depositing images
>To: CCP4BB@JISCMAIL.AC.UK
>
>Dear Enrico, Frank and colleagues,
>
> I am glad to have suggested that everyone's views on this issue should
>be aired out on this BB rather
Dear Enrico, Frank and colleagues,
I am glad to have suggested that everyone's views on this issue should
be aired out on this BB rather than sent off-list to an IUCr committee
member: this is much more interactive and thought-provoking.
There would seem to be clear biases in some of t
given that:
- storage is becoming cheaper exponentially,
- computer power is increasing exponentially,
I think there is no reason to not store all images used for solving a structure
- linked to the pdb entry and properly annotated with beam centre, lambda,
pixel order and all other necessary pro
Dear Peter,
How many crystallographers does it take to transform bad data into good
data?
None, you need a modeller. Only a modeller can give you a structure with
perfect
geometry. Data just introduces experimental errors into what would
otherwise be a perfect
structure.
If you have good
cp4bb] IUCr committees, depositing images
On the deposition of raw data:
Committees, wherever you are!
I guess that abstaining from storing the raw diffraction data in the form of
frames is not very wise
whatever its size is. I regret that for some PDB entries I do not have
diffraction data (needle
Dear Enrico,
Please don't get me wrong: what you are saying is not incorrect, but it
is only half the story.
On Tue, 2011-10-18 at 15:13 +0200, Enrico Stura wrote:
> With improving techniques, we should always be making progress!
Yes, of course!
> If we are trying to answer a biological questio
For the record, the amount of disk storage space per unit cost has doubled
every 14 months for the last 30 years. It's an exponential relationship:
www.mkomo.com/cost-per-gigabyte
So data generated at a very high rate today, will be trivial to store in the
near future. That's not to say it is
C.UK] on behalf of Felix
Frolow [mbfro...@post.tau.ac.il]
Sent: Tuesday, October 18, 2011 1:40 PM
To: CCP4BB@JISCMAIL.AC.UK
Subject: Re: [ccp4bb] IUCr committees, depositing images
I could not agree less. There is constant development of the software
for refinement that allow to do thing
Frolow
[mbfro...@post.tau.ac.il]
Sent: Tuesday, October 18, 2011 1:40 PM
To: CCP4BB@JISCMAIL.AC.UK
Subject: Re: [ccp4bb] IUCr committees, depositing images
I could not agree less. There is constant development of the software for
refinement that allow to do things that were not
possible or were not ne
I could not agree less. There is constant development of the software for
refinement that allow to do things that were not
possible or were not necessary in the past such as intelligent refinement of
occupancies of mutually exclusive sites, entities and conformations.
I frequently remeasure lyso
Perhaps this kind of discussion that should be continued on the IUCr
forums, once people have had a chance to register. However, to answer
these particular points:
On Tue, 2011-10-18 at 10:52 +, Chris Morris wrote:
> Some crystals are hard to make, so storing all the data the best way to get
Some crystals are hard to make, so storing all the data the best way to get
reproducibility. On the other hand, no one needs more images of lysozyme. So
using the same standard for every deposition doesn't sound right.
The discussion should be held on the basis of overall cost to the research
b
Dear Colleagues,
Following on from my posting to the CCP4bb of yesterday an IUCr Forum
has been set up for Public input on the diffraction data deposition
future. Thus this Forum will record an organised set of inputs for
future reference. Instructions on how to register at this Forum can be
found
We overestimate the value of individual structures because we're human :)
If a problem is important enough that one structure makes or breaks the
case, a sensible thing to do would be to get more structures and strive to
obtain some other flavor of pertinent information by methods that are
unlikel
On 17/10/2011 01:52, Wladek Minor wrote:
Frank,
This is serious problem for biologists. There is a structure with
ligand. The same data were re-interpreted and people did not find the
ligand. This re-interpretation is not really valid until we will look
into diffraction data. Biologist lost
I'm simultaneously embarrassed to have not read the thread to completion before
replying, and also totally pumped that I would answer a question the same way
as Bernhard Rupp!
On Oct 16, 2011, at 7:34 PM, Bernhard Rupp (Hofkristallrat a.D.) wrote:
> Ø Not if you are interested in scattering th
Ø Not if you are interested in scattering that falls between reciprocal
lattice maxima, or if you want to preserve the possibility of applying
future data reduction packages.
Yep, this is exactly what I expressed in my original statement:
diffuse solvent contributions, commensurate and
On Oct 16, 2011, at 4:18 PM, Bosch, Juergen wrote:
> The rest is comparable to a collection of stamps, although with the benefit
> as BR mentioned of adding an additional hurdle/layer to falsifying structures.
Not if you are interested in scattering that falls between reciprocal lattice
maxima
Ø Do you mean to reprocess determination of I and sig(I) from the
diffraction images automatically??? Or just to get an access to the raw
data?
Reprocessing the images with the to-be-developed new software that will
process the information in the data in full, and then using the
new-and-improv
On Oct 16, 2011, at 4:42 PM, Guenter Fritz wrote:
In the end for the determination of the structure only a few datasets
will be necessary. This means maybe 0.5 Tb of compressed data to
deposit. I don't think this is too much.
I think those 0.5 TB will be not essential, the more interesting data
om: Bosch, Juergen [mailto:jubo...@jhsph.edu]
> Sent: Sunday, October 16, 2011 1:31 PM
> To: hofkristall...@gmail.com
> Cc: CCP4BB@JISCMAIL.AC.UK
> Subject: Re: [ccp4bb] IUCr committees, depositing images
>
> Wasn't that already implemented in Phenix ?
>
> Jü
As in reprocess completely from images, I meant.
From: Bosch, Juergen [mailto:jubo...@jhsph.edu]
Sent: Sunday, October 16, 2011 1:31 PM
To: hofkristall...@gmail.com
Cc: CCP4BB@JISCMAIL.AC.UK
Subject: Re: [ccp4bb] IUCr committees, depositing images
Wasn't that already implement
On the technical feasibility of storage of original data:
Sure, running Pilatus for an olympic record, we will go home with several T of
data after 24 h (will we?).
Yes, we do already. I just checked the number of images from PILATUS 6M
we have collected so far this year : ca. 1.7 millions.
Wasn't that already implemented in Phenix ?
Jürgen
On Oct 16, 2011, at 4:20 PM, Bernhard Rupp (Hofkristallrat a.D.) wrote:
REPROCESS_PDB
..
Jürgen Bosch
Johns Hopkins University
Bloomberg School of Public Health
Department of Biochemistry & Molecular Biology
Johns Hopkins M
Message-
From: CCP4 bulletin board [mailto:CCP4BB@JISCMAIL.AC.UK] On Behalf Of Frank
von Delft
Sent: Sunday, October 16, 2011 12:00 PM
To: CCP4BB@JISCMAIL.AC.UK
Subject: Re: [ccp4bb] IUCr committees, depositing images
One other question (for both key issues described): what exactly is the
problem
On the deposition of raw data:
Committees, wherever you are!
I guess that abstaining from storing the raw diffraction data in the form of
frames is not very wise
whatever its size is. I regret that for some PDB entries I do not have
diffraction data (needless to say that authors
do not submitted
One other question (for both key issues described): what exactly is the
problem the committees are aiming to address?
Because I can't help noticing that Tom's email did not spark an on-list
discussion; do people actually feel either are issues? Isn't the more
burning problem how best to use
On the deposition of raw data:
I recommend to the committee that before it convenes again, every member
should go collect some data on a beamline with a Pilatus detector [feel
free to join us at Diamond]. Because by the probable time any
recommendations actually emerge, most beamlines will ha
76 matches
Mail list logo
