I think it important to point out that despite the subject line, Dr.
Scott's statement was:
"I think they process a bit faster too"
Strangely enough, this has not convinced me to re-format my RAID array
with an new file system nor re-write all my software to support yet
another new file format. I guess I am just lazy that way. Has anyone
measured the speed increase? Have macs become I/O-bound again?
In any case, I think it is important to remember that there are good
reasons for leaving image file formats uncompressed. Probably the most
important is the activation barrier to new authors writing new programs
that read them. "fread()" is one thing, but finding the third-party
code for a particular compression algorithm, navigating a CVS repository
and linking to a library are quite another! This is actually quite a
leap for those of us who never had any formal training in computer
science. Personally, I still haven't figured out how to read pck
images, as it is much easier to write "jiffy" programs for uncompressed
data. For example, if all you want to do is extract a group of pixels
(such as a spot), then you have to decompress the whole image! In
computer speak: fseek() is rendered useless by compression. This could
be why Mar opted not to use the pck compression for their newer
CCD-based detectors?
That said, compressed file systems do appear particularly attractive if
space is limiting. Apparently HFS can do it, but what about other
operating systems? Does anyone have experience with a Linux file system
that both supports compression and doesn't get corrupted easily?
-James Holton
MAD Scientist
Graeme Winter wrote:
Hi David,
If the data compression is carefully chosen you are right: lossless
jpeg2000 compression on diffraction images works very well, but is a
spot slow. The CBF compression using the byte offset method is a
little less good at compression put massively faster... as you point
out, this is the one used in the pilatus images. I recall that the
.pck format used for the MAR image plates had the same property - it
was quicker to read in a compressed image that the raw equivalent.
So... once everyone is using the CBF standard for their images, with
native lossless compression, it'll save a fair amount in disk space
(=£/$), make life easier for people and - perhaps most importantly -
save a lot of data transfer time.
Now the funny thing with this is that if we compress the images before
we store them, the compression implemented in the file system will be
less effective... oh well, can't win em all...
Cheers,
Graeme
2009/9/18 Waterman, David (DLSLtd,RAL,DIA) <david.water...@diamond.ac.uk>:
Just to comment on this, my friend in the computer game industry insists
that compression begets speed in almost all data handling situations.
This will be worth bearing in mind as we start to have more fine-sliced
Pilatus 6M (or similar) datasets to deal with.
Cheers,
David.
-----Original Message-----
From: CCP4 bulletin board [mailto:ccp...@jiscmail.ac.uk] On Behalf Of
William G. Scott
Sent: 17 September 2009 22:48
To: CCP4BB@JISCMAIL.AC.UK
Subject: [ccp4bb] I compressed my images by ~ a factor of two, and they
load and process in mosflm faster
If you have OS X 10.6, this will impress your friends and save you some
disk space:
% du -h -d 1 mydata
3.5G mydata
mv mydata mydata.1
sudo ditto --hfsCompression mydata.1 mydata rm -rf mydata.1
% du -h -d 1 mydata
1.8G mydata
This does hfs filesystem compression, so the images are still recognized
by mosflm, et al. I think they process a bit faster too, because half
the information is packed into the resource fork.
This e-mail and any attachments may contain confidential, copyright and or
privileged material, and are for the use of the intended addressee only. If you
are not the intended addressee or an authorised recipient of the addressee
please notify us of receipt by returning the e-mail and do not use, copy,
retain, distribute or disclose the information in or attached to the e-mail.
Any opinions expressed within this e-mail are those of the individual and not
necessarily of Diamond Light Source Ltd.
Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments
are free from viruses and we cannot accept liability for any damage which you
may sustain as a result of software viruses which may be transmitted in or with
the message.
Diamond Light Source Limited (company no. 4375679). Registered in England and
Wales with its registered office at Diamond House, Harwell Science and
Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom
