Dear Michael,
On Nov 19, 9:34 pm, mabshoff <[EMAIL PROTECTED]
dortmund.de> wrote:
> I think it is much more important to understand what the intended use
> here is instead of arguing about the finer details of compression.
Good point, although I think that the speed/memory issues pointed out
by David are helpful to make a decision.
The intended use is:
- I am computing cohomology rings, and during the computation some
executables are called that produce a lot(!) of data, stored in files
F1,F2,....
- When I implemented pickling/unpickling, I first tried to save the
cohomology ring (including data that are in F1,F2,...) into a single
file. However, it turned out to be too slow and to consume too much
memory, such that it became unfeasible in some examples.
- Hence, when I save a cohomology ring now, I just provide information
on how to reconstruct the ring from F1,F2,..., in particular it tells
the location of F1,F2,...
- Advantage: It is fast and works well, as long as you stay on a
single machine.
- Disadvantage: Clearly it isn't portable. Even if you move the pickle
plus all F1,F2,... manually on a different machine, you couldn't
reconstruct the ring, because the location of the F1,F2,... changed (-
> broken paths).
Hence, I made an export and an import method:
- export puts all data needed to reconstruct the ring into an archive
file.
- One may transfer the archive file to a different location. The
import method unpacks it and takes care that the unpacked F1,F2,...
are put correctly relative to the new location.
So far, I could live without import/export. But my project eventually
aims at publishing the computational results in a data base.
Memory is an issue. The data base will comprise more than 2400
cohomology rings, and a GNU zipped tar file for one of the biggest
examples is 65 MB.
And currently I am trying to compute a ring that occupies a total of
74 GB (uncompressed) on the disk - it isn't finished yet...
> I would always suggest tar.gz since bz2 support for example is not build
> in for non-GNU tar.
Does the system-tar really play a role if one can use the Python
libraries for data compression? I understood that the bz2 package
would do the work, even if there is no GNU tar on the system.
Cheers,
Simon
--~--~---------~--~----~------------~-------~--~----~
To post to this group, send email to sage-support@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at
http://groups.google.com/group/sage-support
URLs: http://www.sagemath.org
-~----------~----~----~----~------~----~------~--~---
