I was importing into sleepycat using standard elephant routines. I am
not aware of an 'import mode' for sleepycat, but I will look into that
when I have a chance. Another consideration using sleepycat is that
using BTrees with a large working set demands large amounts of memory
relative to a Hash representation. I am unfamiliar with the internals
of elephant and sleepycat, but it feels like the basic access method is
restricting performance, which seems to be described here:
http://www.sleepycat.com/docs/gsg/C/accessmethods.html#BTreeVSHash
My problem so far has been importing the data, which goes very fast
until sleepycat requires extensive disk access. The in-memory rate is
reasonable and would complete in a few hours. However, once disk
operations begin the import speed suggests it would take many days to
complete. I have yet to perform extensive benchmarks, but I estimate
the instantiation rate shifts from 1800 persistent class instantiations
/second to 120 / s.
here are the two approaches that I hypothesize may help performance. I
am admittedly unaware of innards of the two systems in question, so you
expert developers will know best. If either sounds appropriate or you
envision another possibility for allowing this kind of scaling, I will
look into implementing such a system.
1. decreasing the size of the working set is one possibility for
decreasing run-time memory requirements and disk access. I'm not sure
how the concept of a 'working set' translates from the sleepycat world
to the elephant world, but perhaps you do.
2. using a Hash instead of a BTree in the primary database? I am
unsure what this means for elephant.
In the mean time I will depart from the every-class-is-persistent
approach and also use more traditional data structures.
Thanks again,
Red Daly
Robert L. Read wrote:
Yes, it's amusing.
In my own work I use the Postgres backend; I know very little about
SleepyCat. It seems
to me that this is more of a SleepyCat issue, then an Elephant issue.
Perhaps you should
ask the SleepyCat list?
Are you importing things into SleepyCat directly in the correct
serialization format that
they can be read by Elephant? If so, I assume it is just a question
of solving the SleepyCat
problems.
An alternative would be to use the SQL-based backend. However, I
doubt this will solve
your problem, since at present we (well, I wrote it) use a very
inefficient serialization scheme
for the SQL-based backend that base64 encodes everything. This had
the advantage that
it makes it work trouble-free with different database backends, but
could clearly be improved upon.
However, it is more than efficient enough for all my work, and at
present nobody is clamoring
to have it improved.
Is your problem importing the data or using it once it is imported?
It's hard for me to imagine
a problem so large that even the import time is a problem --- suppose
it takes 24 hours --- can
you not afford to pay that?
A drastic measure and potentially expensive measure would be to switch
to a 64-bit architecture
with a huge memory. I intend to do that when forced by performance
issues in my own work.
On Tue, 2006-10-10 at 00:46 -0700, Red Daly wrote:
I will be running experiments in informatics and modeling in the future
that may contain (tens or hundreds of) millions of objects. Given the
ease of use of elephant so far, it would be great to use it as the
persistent store and avoid creating too many custom data structures.
I have recently run up against some performance bottlenecks when using
elephant to work with very large datasets (in the hundreds of millions
of objects). Using SleepyCat, I am able to import data very quickly
with a DB_CONFIG file with the following contents:
set_lk_max_locks 500000
set_lk_max_objects 500000
set_lk_max_lockers 500000
set_cachesize 1 0 0
I can import data very quickly until the 1 gb cache is too small to
allow complete in-memory access to the database. at this point it seems
that disk IO makes additional writes happen much slower. (I have also
tried increasing the 1 gb cache size, but the database fails to open if
it is too large--e.g. 2 gbs. I have 1.25 gb physical memory and 4 gb
swap, so the constraint seems to be physical memory.) the max_lock,
etc. lines allow transactions to contain hundreds of thousands of
individual locks, limiting the transaction throughput bottleneck
What are the technical restrictions on writing several million objects
to the datastore? Is it feasible to create a batch import feature to
allow large datasets to be imported using reasonable amounts of memory
for a desktop computer?
I hope this email is at least amusing!
Thanks again,
red daly
_______________________________________________
elephant-devel site list
elephant-devel@common-lisp.net <mailto:elephant-devel@common-lisp.net>
http://common-lisp.net/mailman/listinfo/elephant-devel
------------------------------------------------------------------------
_______________________________________________
elephant-devel site list
elephant-devel@common-lisp.net
http://common-lisp.net/mailman/listinfo/elephant-devel
_______________________________________________
elephant-devel site list
elephant-devel@common-lisp.net
http://common-lisp.net/mailman/listinfo/elephant-devel
