Dear James and other skeptics, pessimistic nay-sayers, nattering
nabobs of negativism, and incorrigible vultures of ill-omen:
This sort of compression is a bit different from most, in that is is
both lossless and transparent, and it is only possible with respect to
the file system (HFS+ in this case, and I think zfs, which is better,
and which will eventually replace HFS+ on Apple).
So let's say I have a bunch of images in a directory, 5 GB, and I
create a tarball on some standard file system, that is say 3.5 GB.
(I'm guessing at numbers).
I can use HFS compression on the images, essentially storing about 2/3
of the information in the resource fork. The directory size becomes
about 2.5 GB. The data are completely unchanged. They are simply
stored more efficiently.
Now I make a tarball, using an HFS+ competent version of tar. The
size of that tarball is 3.5 GB. Still. Why? Because the resource
forks are paired with the data forks, and no data is lost.
If I now expand this tarball of the HFS compressed images on some
other unix file system, the resulting images are identical in size and
content to the uncompressed images from my reference (control)
tarball. No data is lost.
So this form of compression is fundamentally different in the sense
that it doesn't really DO anything to the data, except arrange it on
the filesystem more efficiently. You don't have to re-write any
software. The standard unix tools work on the file in the standard way.
ls -l even reports that the file is of the original, uncompressed
size, since that is what is "sees." Same with mosflm. That is why I
had to use du to show the effect.
eg:
zsh-% sudo ditto --hfsCompress coot-real coot-real-hfscompressed
zsh-% ls -l coot-real*
-rwxr-xr-x 1 root admin 5.0M Sep 14 12:53 coot-real*
-rwxr-xr-x 1 root admin 5.0M Sep 18 21:57 coot-real-hfscompressed*
zsh-% du -h -d 0 coot-real*
5.0M coot-real
1.8M coot-real-hfscompressed
To use a crude analogy, it's kind of as if ls operates in the total
space and du operates in the base space of a fiber bundle.
Except different.
;)
HTH,
Bill
On Sep 18, 2009, at 1:20 PM, James Holton wrote:
I think it important to point out that despite the subject line, Dr.
Scott's statement was:
"I think they process a bit faster too"
Strangely enough, this has not convinced me to re-format my RAID
array with an new file system nor re-write all my software to
support yet another new file format. I guess I am just lazy that
way. Has anyone measured the speed increase? Have macs become I/O-
bound again?
In any case, I think it is important to remember that there are good
reasons for leaving image file formats uncompressed. Probably the
most important is the activation barrier to new authors writing new
programs that read them. "fread()" is one thing, but finding the
third-party code for a particular compression algorithm, navigating
a CVS repository and linking to a library are quite another! This
is actually quite a leap for those of us who never had any formal
training in computer science. Personally, I still haven't figured
out how to read pck images, as it is much easier to write "jiffy"
programs for uncompressed data. For example, if all you want to do
is extract a group of pixels (such as a spot), then you have to
decompress the whole image! In computer speak: fseek() is rendered
useless by compression. This could be why Mar opted not to use the
pck compression for their newer CCD-based detectors?
That said, compressed file systems do appear particularly attractive
if space is limiting. Apparently HFS can do it, but what about
other operating systems? Does anyone have experience with a Linux
file system that both supports compression and doesn't get corrupted
easily?
-James Holton
MAD Scientist
Graeme Winter wrote:
Hi David,
If the data compression is carefully chosen you are right: lossless
jpeg2000 compression on diffraction images works very well, but is a
spot slow. The CBF compression using the byte offset method is a
little less good at compression put massively faster... as you point
out, this is the one used in the pilatus images. I recall that the
.pck format used for the MAR image plates had the same property - it
was quicker to read in a compressed image that the raw equivalent.
So... once everyone is using the CBF standard for their images, with
native lossless compression, it'll save a fair amount in disk space
(=£/$), make life easier for people and - perhaps most importantly -
save a lot of data transfer time.
Now the funny thing with this is that if we compress the images
before
we store them, the compression implemented in the file system will be
less effective... oh well, can't win em all...
Cheers,
Graeme
2009/9/18 Waterman, David (DLSLtd,RAL,DIA) <david.water...@diamond.ac.uk
>:
Just to comment on this, my friend in the computer game industry
insists
that compression begets speed in almost all data handling
situations.
This will be worth bearing in mind as we start to have more fine-
sliced
Pilatus 6M (or similar) datasets to deal with.
Cheers,
David.
-----Original Message-----
From: CCP4 bulletin board [mailto:ccp...@jiscmail.ac.uk] On Behalf
Of
William G. Scott
Sent: 17 September 2009 22:48
To: CCP4BB@JISCMAIL.AC.UK
Subject: [ccp4bb] I compressed my images by ~ a factor of two, and
they
load and process in mosflm faster
If you have OS X 10.6, this will impress your friends and save you
some
disk space:
% du -h -d 1 mydata
3.5G mydata
mv mydata mydata.1
sudo ditto --hfsCompression mydata.1 mydata rm -rf mydata.1
% du -h -d 1 mydata
1.8G mydata
This does hfs filesystem compression, so the images are still
recognized
by mosflm, et al. I think they process a bit faster too, because
half
the information is packed into the resource fork.
This e-mail and any attachments may contain confidential,
copyright and or privileged material, and are for the use of the
intended addressee only. If you are not the intended addressee or
an authorised recipient of the addressee please notify us of
receipt by returning the e-mail and do not use, copy, retain,
distribute or disclose the information in or attached to the e-mail.
Any opinions expressed within this e-mail are those of the
individual and not necessarily of Diamond Light Source Ltd.
Diamond Light Source Ltd. cannot guarantee that this e-mail or any
attachments are free from viruses and we cannot accept liability
for any damage which you may sustain as a result of software
viruses which may be transmitted in or with the message.
Diamond Light Source Limited (company no. 4375679). Registered in
England and Wales with its registered office at Diamond House,
Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11
0DE, United Kingdom
