Rob Coops wrote:
Dear list,
I have a question to anyone who has experience with the following situation.
I have a very complex array build from hashes containing arrays that contain
hashes etc. I would like to store the whole thing on a disk, both to save on
memory usage and to make it easier to separate this structure from the
actual code.
First off: Sounds like a good idea ;-)
The trick is that I would not like to have to read back the entire array in
one go (it is multiple megabytes already and will only grow in the future).
So I would like to be able to access the array elements and only read one
element at a time rather then the entire thing in one go, I would also like
to be able to store the file in an encrypted format but that is more of a
fun thing then a real need.
Sounds like this is not a one-off script. I know thats not the easiest
thing to do, but you should look into using a database with a few
well-defined tables.
I recommend PostgreSQL (personal favourite), but MySQL or even maybe
SQLite would do the trick.
Perls DBI/DBD modules are very good and very well documented. If you
choose PostgreSQL (www.postgresql.org), all the documentation is online
as well.
With a database, you don't have to handle all that data storage and
retrieval, can run multiple programs on the data simultaniously. And
best of all: Unless you made a really *big* blunder in your table
definitions, you probably get better performance than a
reinvent-the-wheel solution could get you.
If you choose to implement the data handling on a file base, you also
have to worry about broken files if your program crashes during write
operations (databases already have rollback-logging/write-ahead-logs
implemented).
Could someone please advise me what module would be best for
this resulting in the smallest possible data file while still offering a
reasonable speed. A direct tie between the disk and the
applications operation is not fast enough... as an 'root' element in the
array gets read once but the data structure it contains gets used hundreds
or thousands of times during the execution of the program.
This *really* depends on your data structure. Can you give us an example?
BTW, you can also use Storable to store small to medium data structures
in a database text field by freezing/thawing and base64-encoding them,
something like this:
use Storable qw(freeze thaw);
use MIME::Base64;
sub dbfreeze {
my ($data) = @_;
return encode_base64(freeze($data), "");
}
sub dbthaw {
my ($data) = @_;
return thaw(decode_base64($data));
}
LLAP & LG
Rene
--
To unsubscribe, e-mail: beginners-unsubscr...@perl.org
For additional commands, e-mail: beginners-h...@perl.org
http://learn.perl.org/
