Le 08/11/11 10:15, Kay Diederichs a écrit : > Hi James, > > I see no real need for lossy compression datasets. They may be useful > for demonstration purposes, and to follow synchrotron data collection > remotely. But for processing I need the real data. It is my experience > that structure solution, at least in the difficult cases, depends on > squeezing out every bit of scattering information from the data, as much > as is possible with the given software. Using a lossy-compression > dataset in this situation would give me the feeling "if structure > solution does not work out, I'll have to re-do everything with the > original data" - and that would be double work. Better not start going > down that route. > > The CBF byte compression puts even a 20bit detector pixel into a single > byte, on average. These frames can be further compressed, in the case of > Pilatus fine-slicing frames, using bzip2, almost down to the level of > entropy in the data (since there are so many zero pixels). And that > would be lossless. > > Storing lossily-compressed datasets would of course not double the > diskspace needed, but would significantly raise the administrative burdens. > > Just to point out my standpoint in this whole discussion about storage > of raw data: > I've been storing our synchrotron datasets on disks, since 1999. The > amount of money we spend per year for this purpose is constant (less > than 1000€). This is possible because the price of a GB disk space drops > faster than the amount of data per synchrotron trip rises. So if the > current storage is full (about every 3 years), we set up a bigger RAID > (plus a backup RAID); the old data, after copying over, always consumes > only a fraction of the space on the new RAID. > > So I think the storage cost is actually not the real issue - rather, the > real issue has a strong psychological component. People a) may not > realize that the software they use is constantly being improved, and > that needs data which cover all the corner cases; b) often do not wish > to give away something because they feel it might help their > competitors, or expose their faults. > > best, > > Kay (XDS co-developer) >
Hi Kay and others, I completely agree with you. Datalove, <3 :-) -- Miguel Architecture et Fonction des Macromolécules Biologiques (UMR6098) CNRS, Universités d'Aix-Marseille I & II Case 932, 163 Avenue de Luminy, 13288 Marseille cedex 9, France Tel: +33(0) 491 82 55 93 Fax: +33(0) 491 26 67 20 mailto:miguel.ortiz-lombar...@afmb.univ-mrs.fr http://www.afmb.univ-mrs.fr/Miguel-Ortiz-Lombardia
