Hi Alexander and Gordon, merge_index is inspired by bitcask, but it's not a strict derivative. It does *not* keep the keydir structure in memory, however it keeps some hints in memory. The memory usage does grow with the number of terms, albeit slowly, since it is mostly keeping signatures and offset information, and the hints are compressed.
To get into the details, merge_index maintains one in-memory offset entry for (approximately) every 64kb block of index. The offset entry consists of: - A few bytes for bookkeeping. - 200 bytes of fixed overhead for a bloom filter. - A short entry for each term header in the block consisting of: - Offset and count information for that term. - Two highly compressible signatures. Then the whole structure is compressed. In practice, memory usage has not been a problem even for large vocabularies. Best, Rusty On Sat, Jan 22, 2011 at 11:36 AM, Alexander Sicular <sicul...@gmail.com>wrote: > Without knowing exactly, I'm gonna go with yes. I happen to be under the > impression that the merge_index_backend is a bitcask derivative. But I would > love to hear otherwise from someone @basho. > > -Alexander Sicular > > @siculars > > On Jan 22, 2011, at 9:50 AM, Gordon Tillman wrote: > > > Greetings All, > > > > It is my understanding the only backend that is compatible with Riak > Search indexes is the merge_index_backend. > > > > I am wondering if merge_index backend has a similar memory footprint as > bitcask; i.e., must the keydir structure for merge_index fit entirely in RAM > as is the case with bitcask? > > > > Many thanks! > > > > --gordon > > _______________________________________________ > > riak-users mailing list > > riak-users@lists.basho.com > > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > > > _______________________________________________ > riak-users mailing list > riak-users@lists.basho.com > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >
_______________________________________________ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
