This last, very important point made by Graeme gives me the chance to plug a satellite workshop at the IUCr Congress (end of August, so fingers crossed that this will go ahead) on "MX raw image data formats, metadata and validation". This is organized by Herbert Bernstein, Loes Kroon-Batenburg, Brian McMahon and myself, and will take place on 22 Aug from morning till 3 pm.
Why is this relevant? Over the past few years, many beamline scientists from synchrotrons (and XFELs) the world over have worked hard to turn the perception of a lack of metadata into a transcription of essential metadata items in a language that can be saved with current data formats (HDF5 and CBF), can be used for archiving and retrieval of data, and is understood by automatic processing pipelines. We like to call this the Gold Standard. The Gold Standard demands the beam center is recorded for every experiment. It also insists on synchrotron and beamline, items that are curiously absent from a lot of datasets. In the Gold Standard, you'll find a full description of the diffraction experiment and won't have to guess anymore what the rotation axis is and how the detector is positioned. The purpose of the Prague workshop is to make the Gold Standard better known to a wider audience, explain what's in it and how different synchrotrons are applying it, and discuss how to expand this to chemical crystallography. The program is still a work in progress and the website rather austere ( https://www.iucr.org/resources/data/commdat/prague-workshop-mx-raw-data), but mark your calendars if you're interested. We'll have a great line-up of speakers and ample opportunity for discussions. As the workshop will take place in the conference center, it will be only a short walk to the opening ceremony later that day. All best. Andreas (wearing my DECTRIS hat). On Thu, Mar 19, 2020 at 10:08 AM Winter, Graeme (DLSLtd,RAL,LSCI) < graeme.win...@diamond.ac.uk> wrote: > Hi Folks, > > Quick note on "Like a super raw image diet-plan for the incoming summer” - > the bslz4 compression used with Eiger works really well, and the data are > pretty close to entropy limit in terms of size - which means if you measure > your data carefully (i.e. low background etc.) you can realistically have a > full Eiger data set which will fit on a DVD - > > Grey-Area i04-ins :) $ du -ach insu_d200_1*h5 > 2.5G insu_d200_1_000001.h5 > 1.9G insu_d200_1_000002.h5 > 92K insu_d200_1_master.h5 > 138M insu_d200_1_meta.h5 > 4.0K insu_d200_1_meta_pack.h5 > 4.5G total > > which I think is pretty modest storage wise - this is 1,800 image Eiger > 16M data set so pretty realistic. > > No matter how much data we produce, the particle physicists have us beat, > and I would wager that netflix is a couple of orders of magnitude worse > than MX for the environment. > > Also, it costs a lot more CO2 to produce the data than to store it - > synchrotrons don’t run on wind turbines ;-) > > I agree with Clemens - if we are serious about revisiting the data, you > have to have access to the actual data rather than a description of the > data (which is essentially what an MTZ file is) > > A small administrative note for those uploading to zenodo etc - > > if you are uploading HDF5 as above, this works fine > if you are uploading CBF images, would suggest you gzip / bzip2 compress > them (pick one) and then tar up each sweep independently before upload > > - makes fetching the data back down again efficient. Zenodo does not do > well with many many small files (I have discussed this with the zenodo > developers - it is a reasonable design choice) > > Finally I would ask if you do upload raw data please also upload the > beamline parameters you used e.g. the beam centre from a white board or > whatever - if it’s not in the headers it is not known > > All the best Graeme > > > On 19 Mar 2020, at 08:47, Julien Cappèle < > julien.capp...@univ-lorraine.fr> wrote: > > > > Hi everyone, > > > > There are some very interesting ideas. > > > > Though I agree with you Clement that raw images are amazing to work with > as you can use any software you are confortable with, we cannot forget that > depositing several TB of data for each lab would be bad for ecological > reason. And because detectors are always improving (thank you all!), size > of data will increase exponentially. > > > > Correct me if I'm wrong, as I am not that familiar with the way that > integration of raw images works: > > > > Could it be possible for a new/already existing software to store > reflections (area, intensity from center to border, position x/y on the > image, and information of the image) in a lightweight and text only file ? > Possibly a new format to be used for integration ? > > > > While those text files would be heavy, they'd be still lighter than raw > images and the whole useless white space they carry with them between > reflections. > > > > If not possible, could we imagine to eliminate all the unused space on > these ? Like a super raw image diet-plan for the incoming summer ! > > > > Best regards, > > > > Julien CAPPELE > > Université de Lorraine, Nancy, France > > PhD student - 2nd Year > > > > ######################################################################## > > > > To unsubscribe from the CCP4BB list, click the following link: > > https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCP4BB&A=1 > > > -- > This e-mail and any attachments may contain confidential, copyright and or > privileged material, and are for the use of the intended addressee only. If > you are not the intended addressee or an authorised recipient of the > addressee please notify us of receipt by returning the e-mail and do not > use, copy, retain, distribute or disclose the information in or attached to > the e-mail. > Any opinions expressed within this e-mail are those of the individual and > not necessarily of Diamond Light Source Ltd. > Diamond Light Source Ltd. cannot guarantee that this e-mail or any > attachments are free from viruses and we cannot accept liability for any > damage which you may sustain as a result of software viruses which may be > transmitted in or with the message. > Diamond Light Source Limited (company no. 4375679). Registered in England > and Wales with its registered office at Diamond House, Harwell Science and > Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom > > > ######################################################################## > > To unsubscribe from the CCP4BB list, click the following link: > https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCP4BB&A=1 > -- <https://www.dectris.com> Andreas Förster, Ph.D. Application Scientist Crystallography, Area Sales Manager Asia & Pacific Phone: +41 56 500 21 00 | Direct: +41 56 500 21 76 | Email: andreas.foers...@dectris.com DECTRIS Ltd. | Taefernweg 1 | 5405 Baden-Daettwil | Switzerland | www.dectris.com <https://www.facebook.com/pages/Dectris-Ltd/623855944369304> <https://www.linkedin.com/company/5067919> <https://www.linkedin.com/company/5067919> <https://www.linkedin.com/company/5067919> <https://www.linkedin.com/company/5067919>[image: LinkedIn] <https://www.linkedin.com/company/5067919> <https://www.facebook.com/pages/Dectris-Ltd/623855944369304> <https://www.facebook.com/pages/Dectris-Ltd/623855944369304> <https://www.facebook.com/pages/Dectris-Ltd/623855944369304> <https://www.facebook.com/pages/Dectris-Ltd/623855944369304> <https://www.facebook.com/pages/Dectris-Ltd/623855944369304> <https://www.facebook.com/pages/Dectris-Ltd/623855944369304>[image: facebook] <https://www.facebook.com/pages/Dectris-Ltd/623855944369304> <https://twitter.com/DECTRIS_News> <https://twitter.com/DECTRIS_News> <https://twitter.com/DECTRIS_News> *Confidentiality Note: This message is intended only for the use of the named recipient(s)* *and may contain confidential and/or privileged information. If you are not the intended* *recipient, please contact the sender and delete the message. Any unauthorized use of* *the information contained in this message is prohibited.* ######################################################################## To unsubscribe from the CCP4BB list, click the following link: https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCP4BB&A=1
