Ahh good point. I agree it makes sense to make MMapDir's chunking user-controllable. Can you open an issue?
Mike On Mon, Jul 13, 2009 at 5:49 AM, Uwe Schindler<u...@thetaphi.de> wrote: >> On Sun, Jul 12, 2009 at 10:51 AM, eks dev<eks...@yahoo.co.uk> wrote: >> >> > MMapDirectory has support for chunking (Ineteger.MAX_VALUE) anyhow... >> maybe for such cases this threshold can become user settable. I will try >> to experiment with it (I am talking about MMapDirectory -> private final >> int MAX_BBUF = Integer.MAX_VALUE;) >> >> This chunking doesn't reduce virtual memory usage, because the buffers >> are all mapped up front on opening the IndexInput. >> >> It's only done because of a limitation of ByteBuffer (it uses int not >> long to address the values). > > The idea of chunking is because of address space fragmentation. If the Java > VM allocates or frees memory (e.g. when expanding the heap from Xms to Xmx), > the address space of the process gets fragmented. If the index is very > large, the byte buffer must fit in complete as one fragment into the address > space, which may easily fail when you only have 32 bit. > > If the file is mapped in different chunks, each chunk can be somewhere else > in address space (e.g., not only one 1 Gig junk is needed, there can be ten > 100 Meg junks at different places). > > Uwe > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org > For additional commands, e-mail: java-user-h...@lucene.apache.org > > --------------------------------------------------------------------- To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org For additional commands, e-mail: java-user-h...@lucene.apache.org
