I keep meaning to get back to that algorithm and make it a bit more
"slick" so that people don't need all the third-party stuff to use it.
In fact, I have a better idea for how to do it now I'm calling "noise
replacement compression", but that is awaiting something I like to call
"time" which appears to be in increasingly short supply these days.
Enthusiasm for lossy compression has also been somewhat dampened by
things like this:
http://blog.backblaze.com/2013/02/20/180tb-of-good-vibrations-storage-pod-3-0/
That's 180 TB for 2000 USD. Admittedly not exactly the fire-hardened
highly available redundant fault-tolerant "archive" that you might want
to preserve your data for all time. But if you build one of these
things and (ulp) install Windoze on it, then backblaze.com will be your
off-site redundant copy and maintain that copy for only $5/month. Also
has a nice web interface for retrieving your files. Which, I suppose,
you might be able to tell someone at TARDIS the password for the
particular backblaze account on this "pod" machine and make all the
image data you want "freely available" to the crystallographic community.
Of course, if EVERYONE does this, then backblaze might decide to change
their pricing schedule, but it looks like some astronomers have already
done something like this to them and they thought it was cool enough to
put it on their blog. I suppose if it really is a problem for them, they
might be interested in "licensing" my lossy-compression algorithm. ;)
-James Holton
MAD Scientist
On 3/11/2013 7:50 AM, Pete Meyer wrote:
I use bzip2 as well. In addition, I generally store md5sums of the
images (before and after compression) because it's quicker to check
the hash for validity than load up the images - but this may be overkill.
Pete
Graeme Winter wrote:
Dear Eugene,
Personally I have a habit of using bzip2 for archival of data.
Negative points: very slow. Positive points: universally supported,
lossless. I have lots of data. To be honest most of it I keep in the
native format.
I expect to see plenty of comments of lossless vs. lossy compression
now :o)
N.B. well processed unmerged raw-from-integration .x, INTEGRATE.HKL,
mosflm mtz represents pretty good lossy compression.
Long term storage: depends on your definition of long term, which will
also depend on what you want to do with them... I would guess
useful-long-term would correspond to ~ 10 years to ~ 20 years tops. In
the past I have written data to tape which I have never attempted to
recover. Everything I now have on central file servers (raid systems)
on local fast drives and local cheap USB drives which sit unplugged on
my desk. Local fast drives fail (more frequently than the
manufacturers would admit), raids fail (less frequently but more
catastrophically), local cheap USB drives fail. I bank on them not all
failing at the same time :o) esp. the ones I leave unplugged.
The cheap drives store about 2TB in something the size of a paperback
book and are relatively universal with USB connections. But they will
fail one day. You weigh up how sad you will be at the loss against the
cost of being more certain of not losing the data...
Cheerio,
Graeme
On 11 March 2013 08:39, Eugene Osipov <e.m.osi...@gmail.com> wrote:
Hello everyone,
I have a couple of questions about images compression and storage:
1)do someone use it in routine work and does it works well for them?
2)I found this page by google:
http://bl831.als.lbl.gov/~jamesh/lossy_compression/
And want to ask about reports of usage of this program (of course if
someone
already uses it)
3)Is there any advice for long-term diffraction image storage?
Thank you for your attention,
--
Eugene Osipov
Junior Research Scientist
Laboratory of Enzyme Engineering
A.N. Bach Institute of Biochemistry
Russian Academy of Sciences
Leninsky pr. 33, 119071 Moscow, Russia
e-mail: e.m.osi...@gmail.com
