Hi Folks, Quick note on "Like a super raw image diet-plan for the incoming summer” - the bslz4 compression used with Eiger works really well, and the data are pretty close to entropy limit in terms of size - which means if you measure your data carefully (i.e. low background etc.) you can realistically have a full Eiger data set which will fit on a DVD -
Grey-Area i04-ins :) $ du -ach insu_d200_1*h5 2.5G insu_d200_1_000001.h5 1.9G insu_d200_1_000002.h5 92K insu_d200_1_master.h5 138M insu_d200_1_meta.h5 4.0K insu_d200_1_meta_pack.h5 4.5G total which I think is pretty modest storage wise - this is 1,800 image Eiger 16M data set so pretty realistic. No matter how much data we produce, the particle physicists have us beat, and I would wager that netflix is a couple of orders of magnitude worse than MX for the environment. Also, it costs a lot more CO2 to produce the data than to store it - synchrotrons don’t run on wind turbines ;-) I agree with Clemens - if we are serious about revisiting the data, you have to have access to the actual data rather than a description of the data (which is essentially what an MTZ file is) A small administrative note for those uploading to zenodo etc - if you are uploading HDF5 as above, this works fine if you are uploading CBF images, would suggest you gzip / bzip2 compress them (pick one) and then tar up each sweep independently before upload - makes fetching the data back down again efficient. Zenodo does not do well with many many small files (I have discussed this with the zenodo developers - it is a reasonable design choice) Finally I would ask if you do upload raw data please also upload the beamline parameters you used e.g. the beam centre from a white board or whatever - if it’s not in the headers it is not known All the best Graeme > On 19 Mar 2020, at 08:47, Julien Cappèle <julien.capp...@univ-lorraine.fr> > wrote: > > Hi everyone, > > There are some very interesting ideas. > > Though I agree with you Clement that raw images are amazing to work with as > you can use any software you are confortable with, we cannot forget that > depositing several TB of data for each lab would be bad for ecological > reason. And because detectors are always improving (thank you all!), size of > data will increase exponentially. > > Correct me if I'm wrong, as I am not that familiar with the way that > integration of raw images works: > > Could it be possible for a new/already existing software to store reflections > (area, intensity from center to border, position x/y on the image, and > information of the image) in a lightweight and text only file ? Possibly a > new format to be used for integration ? > > While those text files would be heavy, they'd be still lighter than raw > images and the whole useless white space they carry with them between > reflections. > > If not possible, could we imagine to eliminate all the unused space on these > ? Like a super raw image diet-plan for the incoming summer ! > > Best regards, > > Julien CAPPELE > Université de Lorraine, Nancy, France > PhD student - 2nd Year > > ######################################################################## > > To unsubscribe from the CCP4BB list, click the following link: > https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCP4BB&A=1 -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom ######################################################################## To unsubscribe from the CCP4BB list, click the following link: https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCP4BB&A=1
