Hmm screen shot didn't make it ... can you post link?

If you are using NRT reader then when a new one is opened, it won't
open new SegmentReaders for all segments, just for newly
flushed/merged segments since the last reader was opened.  So for your
N commit points that you have readers open for, they will be sharing
SegmentReaders for segments they have in common.

How many unique fields are you adding?

Mike McCandless

http://blog.mikemccandless.com


On Wed, Aug 27, 2014 at 7:41 PM, Vitaly Funstein <vfunst...@gmail.com> wrote:
> Mike,
>
> Here's the screenshot; not sure if it will go through as an attachment
> though - if not, I'll post it as a link. Please ignore the altered package
> names, since Lucene is shaded in as part of our build process.
>
> Some more context about the use case. Yes, the terms are pretty much unique;
> the schema for the data set is actually borrowed from here:
> https://amplab.cs.berkeley.edu/benchmark/#workload - it's the UserVisits
> set, with a couple of other fields added by us. The values for the fields
> are generated almost randomly, though some string fields are picked at
> random from a fixed dictionary.
>
> Also, this type of heap footprint might be tolerable if it stayed relatively
> constant throughout the system's life cycle (of course, given the index set
> stays more or less static). However, what happens here is that one
> IndexReader reference is maintained by ReaderManager as an NRT reader. But
> we also would like support an ability to execute searches against specific
> index commit points, ideally in parallel. As you might imagine, as soon as a
> new DirectoryReader is opened at a given commit, a whole new set of
> SegmentReader instances is created and populated, effectively doubling the
> already large heap usage... if there was a way to somehow reuse readers for
> unchanged segments already pooled by IndexWriter, that would help
> tremendously here. But I don't think there's a way to link up the two sets,
> at least not in the Lucene version we are using (4.6.1) - is this correct?
>
>
> On Wed, Aug 27, 2014 at 12:56 AM, Michael McCandless
> <luc...@mikemccandless.com> wrote:
>>
>> This is surprising: unless you have an excessive number of unique
>> fields, BlockTreeTermReader shouldn't be such a big RAM consumer.
>>
>> Bu you only have 12 unique fields?
>>
>> Can you post screen shots of the heap usage?
>>
>> Mike McCandless
>>
>> http://blog.mikemccandless.com
>>
>>
>> On Tue, Aug 26, 2014 at 3:53 PM, Vitaly Funstein <vfunst...@gmail.com>
>> wrote:
>> > This is a follow up to the earlier thread I started to understand memory
>> > usage patterns of SegmentReader instances, but I decided to create a
>> > separate post since this issue is much more serious than the heap
>> > overhead
>> > created by use of stored field compression.
>> >
>> > Here is the use case, once again. The index totals around 300M
>> > documents,
>> > with 7 string, 2 long, 1 integer, 1 date and 1 float fields which are
>> > both
>> > indexed and stored. It is split into 4 shards, which are basically
>> > separate
>> > indices... if that matters. After the index is populated (but not
>> > optimized
>> > since we don't do that), the overall heap usage taken up by Lucene is
>> > over
>> > 1 GB, much of which is taken up by instances of BlockTreeTermsReader.
>> > For
>> > instance for the largest segment in one such an index, the retained heap
>> > size of the internal tree map is around 50 MB. This is evident from heap
>> > dump analysis, which I have screenshots of that I can post here, if that
>> > helps. As there are many segments of various sizes in the index, as
>> > expected, the total heap usage for one shard stands at around 280 MB.
>> >
>> > Could someone shed some light on whether this is expected, and if so -
>> > how
>> > could I possibly trim down memory usage here? Is there a way to switch
>> > to a
>> > different terms index implementation, one that doesn't preload all the
>> > terms into RAM, or only does this partially, i.e. as a cache? I'm not
>> > sure
>> > if I'm framing my questions correctly, as I'm obviously not an expert on
>> > Lucene's internals, but this is going to become a critical issue for
>> > large
>> > scale use cases of our system.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org
>> For additional commands, e-mail: java-user-h...@lucene.apache.org
>>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org
> For additional commands, e-mail: java-user-h...@lucene.apache.org

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-user-h...@lucene.apache.org

Reply via email to