Re: [elephant-devel] working with many millions of objects

2007-02-15 Thread Ian Eslick
Some experiments performed on an old topic... On Oct 11, 2006, at 8:57 PM, Red Daly wrote: I was importing into sleepycat using standard elephant routines. I am not aware of an 'import mode' for sleepycat, but I will look into that when I have a chance. Another consideration using sleepy

Re: [elephant-devel] working with many millions of objects

2006-10-11 Thread Ian Eslick
I don't think that performance of the base Elephant is good enough to justify an every-object-is-persistent model. It's just not seamless enough or efficient enough yet. DCM takes a good step in the performance direction for most applications (single lisp images) and I'm hoping that some of the

Re: [elephant-devel] working with many millions of objects

2006-10-11 Thread Ian Eslick
Can you tell me a little bit about what the import operations look like - that is to say how many objects are created per-transaction, how many slots per created object, etc? Things are only cached in-memory during a transaction. To ensure ACID properties (unless you've turned off synchronization

Re: [elephant-devel] working with many millions of objects

2006-10-11 Thread Robert L. Read
On Wed, 2006-10-11 at 17:57 -0700, Red Daly wrote: I was importing into sleepycat using standard elephant routines. I am not aware of an 'import mode' for sleepycat, but I will look into that when I have a chance. I do not think there is anyway to import things directly and to deal with

Re: [elephant-devel] working with many millions of objects

2006-10-11 Thread Red Daly
I was importing into sleepycat using standard elephant routines. I am not aware of an 'import mode' for sleepycat, but I will look into that when I have a chance. Another consideration using sleepycat is that using BTrees with a large working set demands large amounts of memory relative to a

Re: [elephant-devel] working with many millions of objects

2006-10-10 Thread Robert L. Read
Yes, it's amusing. In my own work I use the Postgres backend; I know very little about SleepyCat.  It seems to me that this is more of a SleepyCat issue, then an Elephant issue.  Perhaps you should ask the SleepyCat list? Are you importing things into SleepyCat directly in the correct seriali

[elephant-devel] working with many millions of objects

2006-10-10 Thread Red Daly
I will be running experiments in informatics and modeling in the future that may contain (tens or hundreds of) millions of objects. Given the ease of use of elephant so far, it would be great to use it as the persistent store and avoid creating too many custom data structures. I have recently