MappedByteBuffer duplicates

2017-02-24 Thread Kameron Cole
We have a Lucene engine that creates MappedByteBuffer objects when creating the Lucene index. I don't know Lucene well enough to know if this standard behavior. The mapped files are being created by Lucene, via the JRE's NIO APIs native file mapping underneath each MappedByteBuffer object. We

RE: MappedByteBuffer duplicates

2017-02-24 Thread Uwe Schindler
Hi, that is not an issue, the duplicates are required for so called IndexInput clones and splices. Every search request will create many of them. But there is no need to worry, they are just thin wrappers - they don't allocate any extra off-heap memory. They are just there to have a separate po

RE: MappedByteBuffer duplicates

2017-02-24 Thread Kameron Cole
Actually, at a certain point, they have crashed the machine. The native file mappings are deallocated (unmapped) by the JVM when the MappedByteBuffers are eligible for garbage collection. The problem we're seeing is that there are thousands of MappedByteBuffers which are not eligible for garba

Re: MappedByteBuffer duplicates

2017-02-24 Thread Greg Bowyer
You may need to enable this https://github.com/apache/lucene-solr/blob/master/lucene/core/src/java/org/apache/lucene/store/MMapDirectory.java#L167 If you are a Sun^H^H^H Oracle JVM. On Fri, Feb 24, 2017, at 11:31 AM, Kameron Cole wrote: > Actually, at a certain point, they have crashed the mac

RE: MappedByteBuffer duplicates

2017-02-24 Thread Uwe Schindler
Hi, You did not give us all information. So I can only give some hints, because there could be multiple causes for your problems. There is for sure no bug in Apache Lucene as there are thousands of Solr and Elasticsearch instances running without such problems. > Actually, at a certain point,