On Thu, Aug 8, 2013 at 11:31 AM, Michael McCandless
wrote:
> A number of users have complained about the apparent RAM usage of
> WeakIdentityMap, and it adds complexity to ByteBufferIndexInput to do
> this tracking ... I think defaulting the unmap hack to off is best for
> users of MMapDir.
>
Fo
Yes, definitely. Our typical setup is 16Gb physical RAM and -Xmx4G per node
(index size is about 1-1.5Gb per node). So there is plenty of room for OS
cache, I guess. I'll take a closer look at the number of major page faults, but
at the moment iostat says that everything is pretty fine.
On the
de
>
>
>> -Original Message-
>> From: Michael McCandless [mailto:luc...@mikemccandless.com]
>> Sent: Thursday, August 08, 2013 2:18 PM
>> To: Lucene Users
>> Subject: Re: WeakIdentityMap high memory usage
>>
>> Thanks for bringing closure.
>>
essage-
> From: Michael McCandless [mailto:luc...@mikemccandless.com]
> Sent: Thursday, August 08, 2013 2:18 PM
> To: Lucene Users
> Subject: Re: WeakIdentityMap high memory usage
>
> Thanks for bringing closure.
>
> Note that you should still run a tight ship, ie don&#
?
>>
>> Uwe
>>
>> -
>> Uwe Schindler
>> H.-H.-Meier-Allee 63, D-28213 Bremen
>> http://www.thetaphi.de
>> eMail: u...@thetaphi.de
>>
>>> -Original Message-
>>> From: Michael McCandless [mailto:luc...@mikemccandle
>> -Original Message-----
>> From: Michael McCandless [mailto:luc...@mikemccandless.com]
>> Sent: Wednesday, August 07, 2013 3:45 PM
>> To: Lucene Users
>> Subject: Re: WeakIdentityMap high memory usage
>>
>> This map is used to track all cloned open file
August 07, 2013 3:45 PM
> To: Lucene Users
> Subject: Re: WeakIdentityMap high memory usage
>
> This map is used to track all cloned open files, which can be a very large
> number over time (each search will create maybe 3 of them).
>
> This is done as a "best effort&quo
This map is used to track all cloned open files, which can be a very
large number over time (each search will create maybe 3 of them).
This is done as a "best effort" to prevent SEGV (JVM dies) if you
accidentally try to use an IndexReader after it was closed, while
using MMapDirectory.
However,
We have upgraded from Lucene 3.6 to 4.4.On the production we faced high minor
GC time. Heap dump showed that one of the biggest objects by size is
org.apache.lucene.util.WeakIdentityMap$IdentityWeakReference. About 11 million
instances with about 377 megabytes of memory in total (this is not eve
