Re: [elephant-devel] Garbage collection problem

2007-09-27 Thread Ian Eslick
On Sep 27, 2007, at 11:59 AM, Ben wrote: You might be better off, performance wise, doing this in a C full-text indexing system and wrapping an interface to it. I hadn't thought of that yet. Can you recommend any? Elephant's use of BDB is designed for certain kind of operations and access

Re: [elephant-devel] Garbage collection problem

2007-09-27 Thread Ben
> > You might be better off, performance > > wise, doing this in a C full-text indexing system and wrapping an > > interface to it. > > I hadn't thought of that yet. Can you recommend any? lucene is good, solar is a web service packaging of it, and there is a port for common lisp (montezuma), ruby

Re: [elephant-devel] Garbage collection problem

2007-09-27 Thread Ian Eslick
On Sep 27, 2007, at 11:46 AM, Chris Laux wrote: Thanks Ian, that is some good analysis. - If you are doing a significant amount of deserialization with lots of threads than you should know that each deserialization requires a call to (with-lock ...) to ensure that the shared pool of buffer

Re: [elephant-devel] Garbage collection problem

2007-09-27 Thread Chris Laux
Thanks Ian, that is some good analysis. > - If you are doing a significant amount of deserialization with lots of > threads than you should know that each deserialization requires a call > to (with-lock ...) to ensure that the shared pool of buffer streams is > thread safe (a problem with elephant

Re: [elephant-devel] Garbage collection problem

2007-09-27 Thread Chris Laux
> This was (sb-sprof:with-profiling (:mode :alloc ...) ...) ?? Yes. I exchanged the order of the two tables, as noted below. And the line formatting is messed up at bit by email. What's wrong with it? >> [NB: the order of the two output tables is reversed for ease of use] SBCL version is 1.0.4

Re: [elephant-devel] Garbage collection problem

2007-09-27 Thread Ian Eslick
On Sep 27, 2007, at 9:39 AM, Chris Laux wrote: How about just giving us a profile of memory usage? The profile is at the bottom of this message, in sbcl's sb-sprof format. If you can't interpret the numbers let me know and I'll dig up the explanation from the web. I'm not really sure how t

Re: [elephant-devel] Garbage collection problem

2007-09-27 Thread Ian Eslick
This was (sb-sprof:with-profiling (:mode :alloc ...) ...) ?? On Sep 27, 2007, at 9:39 AM, Chris Laux wrote: How about just giving us a profile of memory usage? The profile is at the bottom of this message, in sbcl's sb-sprof format. If you can't interpret the numbers let me know and I'll d

Re: [elephant-devel] Garbage collection problem

2007-09-27 Thread Chris Laux
> How about just giving us a profile of memory usage? The profile is at the bottom of this message, in sbcl's sb-sprof format. If you can't interpret the numbers let me know and I'll dig up the explanation from the web. I'm not really sure how to interpret the results, and it might be buggy, becau

Re: [elephant-devel] Garbage collection problem

2007-09-25 Thread Ian Eslick
How about just giving us a profile of memory usage? I don't know what Lisp your are using, but Allegro, for example, has a (prof:with- profiling (:type :space) ...) that gives you a breakdown of memory allocation by function. This would be very helpful. While a great deal of attention has

Re: [elephant-devel] Garbage collection problem

2007-09-25 Thread Pierre THIERRY
Scribit Chris Laux dies 25/09/2007 hora 19:33: > > Could you give a small testcase that demonstrates this? > [...] I don't think a small test case would produce this, but that it > is produced by a fairly large database (GBs) with >>1000 BTrees that > was queried by >10 parallel threads. Could you

Re: [elephant-devel] Garbage collection problem

2007-09-25 Thread Chris Laux
Pierre THIERRY wrote: >> while using Elephant I've noticed that garbage collections now take up >> to 10 minutes with the (current model) CPU at 100% load!? > > Could you give a small testcase that demonstrates this? Sorry about the late reply. I haven't yet managed to reproduce the behaviour ou

Re: [elephant-devel] Garbage collection problem

2007-09-14 Thread Pierre THIERRY
Scribit Chris Laux dies 13/09/2007 hora 15:26: > while using Elephant I've noticed that garbage collections now take up > to 10 minutes with the (current model) CPU at 100% load!? Could you give a small testcase that demonstrates this? Curiously, Pierre -- [EMAIL PROTECTED] OpenPGP 0xD9D50D8A

[elephant-devel] Garbage collection problem

2007-09-13 Thread Chris Laux
Hi all, while using Elephant I've noticed that garbage collections now take up to 10 minutes with the (current model) CPU at 100% load!? This even happens when I end all "client" threads and manually invoke a full GC despite low memory use. I've never seen anything like it before. Does anyone have

Re: [elephant-devel] garbage collection

2007-07-15 Thread Dmitri V. Gorbatovsky
Thanks a lot, Ian. Finally I get it right. I makes a few wrong assumptions about robustness and simplicity of general migrate procedure , consider btrees and objects references. Thanks again, Dmitri On Wednesday 11 July 2007 22:18, Ian Eslick wrote: > Some documentation: > > http://common-lisp.ne

Re: [elephant-devel] garbage collection

2007-07-11 Thread Ian Eslick
Some documentation: http://common-lisp.net/project/elephant/doc/Repository-Migration-and- Upgrade.html#Repository-Migration-and-Upgrade http://common-lisp.net/project/elephant/doc/Migration-and-Upgrading- API.html#Migration-and-Upgrading-API This is the safest route: Start lisp & load ele

Re: [elephant-devel] garbage collection

2007-07-11 Thread Dmitri V. Gorbatovsky
On Wednesday 11 July 2007 04:32, Ian Eslick wrote: > However, the best way to reclaim disk space today is to migrate from > your current BDB store to a new one. This will copy over only those > objects that are globally reachable; namely anything reachable from > the controller-root and all index

Re: [elephant-devel] garbage collection

2007-07-10 Thread Ian Eslick
Dmitri, The latest version in CVS does export a compact function that works on BDB stores, but it only truncates the end of the %ELEPHANT file and isn't very reliable because if you haven't dropped the last object added then it can't truncate at all. If you do drop objects, BDB should reu

[elephant-devel] garbage collection

2007-07-10 Thread Dmitri V. Gorbatovsky
Hello, from grateful user. I find that size of %ELEPHANT file (BDB) only grows with time and transactions. No matter how many objects I delete. Is there are any conventional way to shrink it? Sorry if the question is stupid, Iam not really advanced in Berkeley db usage. Thanks, DG ___