Another note on lots of databases. with-open-store is a pretty expensive operation, so unless you are only opening them infrequently and for batch operation, a fine-grained use of with-open-store is likely to cost you a great deal of performance.
Ian Robert L. Read wrote: > Dear Kevin, > I would not go so far as to see it is outside what you should > expect of Elephant, but > certainly if I had to store thousands of datapoints in conceptual > groups numbering in the > hundreds my first instinct would be to place them all in one database, > and put them > in different btrees there. > Sleepycat should be perfectly happy with millions and millions of > data objects. > I vaguely remember running into a problem similar to what you are > having. > The idiom you are using, the macro "with-open-store", should be > closing everything, > and appears to have the code in place to do the right thing. You are > using it correctly, > at least conceptually. > I cannot guess whether this is a bug in the way that we close > things, or some bug > in the Sleepycat library itself that will be harder to work around. > A heroic programmer would take your code and turn it in to a test, > and add the test > the suite, so that even if there is some limit we can not easily work > around, we can > at least explore the limit. If you will post or send me an example of > that, I will > work it into the test suite. It will be good to make sure that both > the Sleepycat and > SQL implementations work correctly in this respect. > > I don't know anything about your needs or application, but you > certainly should consider > simply using one database, unless there is some obvious reason this is > bad for you. > (In my opinion, the issue of "data ownership" is the only one that > typically drives you > to split a datastore across separate databases: if client X things > they own the data > and want to be able to wipe it out and back it up independent of > client Y, it is often > easiest to use completely separate datastores. But there is not much > of a performance > reason to do so. > > > On Mon, 2006-03-06 at 18:25 -0500, Kevin Y. Kim (Lists) wrote: >> Hi, >> >> (Perhaps I'm going about this all wrong, but...) >> >> I'm trying to populate a large number of Sleepycat databases. >> I've created the necessary underlying directories. >> What I'm attempting is something like this: >> >> (defun populate-db (tree data-pairs) >> (mapcar #'(lambda (p) (setf (get-value (car p) tree) (cadr p))) >> data-pairs)) >> >> (mapcar #'(lambda (db) >> (with-open-store db >> (setf btree (make-btree) >> (add-to-root "tree" btree) >> (populate-db *btree* list-of-data-pairs))) >> list-of-databases) >> >> I have hundreds of databases, with each database having thousands of >> data-pairs. >> >> So, after loading up around 150 databases, I get an error from >> Sleepycat that >> I've got too many databases open. I *assume* that even though I've >> wrapped >> everything in the "with-open-store" call, the databases are not closing >> because I've got a reference to a btree. Is that correct? If so, is >> there an >> explicit way I can ensure that each database is closed before I move >> on to the >> next one? >> >> If anyone has alternate suggestions on how I should store this data, >> feel free to >> pass them along. >> >> Thanks, >> -kevin >> >> _______________________________________________ >> elephant-devel site list >> elephant-devel@common-lisp.net <mailto:elephant-devel@common-lisp.net> >> http://common-lisp.net/mailman/listinfo/elephant-devel >> > ---- > Robert L. Read, PhD read &T > robertlread.net > Consider visiting Progressive Engineering: http://robertlread.net/pe > In Austin: 912-8593 "Think > globally, Act locally." -- RBF > > > ------------------------------------------------------------------------ > > _______________________________________________ > elephant-devel site list > elephant-devel@common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel _______________________________________________ elephant-devel site list elephant-devel@common-lisp.net http://common-lisp.net/mailman/listinfo/elephant-devel
