Please shed more insight into the difference between JVM heap size and the memory size used by Lucene.
What I am getting at is that no matter however much ram I give my apps, Lucene can't utilize it. Is that right? What about the ByteBufferDirectory? Can this specific directory utilize the 2GB memory I grant to the app? On Mon, Jun 4, 2012 at 10:58 PM, Jason Rutherglen < jason.rutherg...@gmail.com> wrote: > If you want the index to be stored completely in RAM, there is the > ByteBuffer directory [1]. Though I do not see the point in putting an > index in RAM, it will be cached in RAM regardless in the OS system IO > cache. > > 1. > https://github.com/elasticsearch/elasticsearch/blob/master/src/main/java/org/apache/lucene/store/bytebuffer/ByteBufferDirectory.java > > On Mon, Jun 4, 2012 at 10:55 AM, Cheng <zhoucheng2...@gmail.com> wrote: > > My indexes are 500MB+. So it seems like that RAMDirectory is not good for > > that big a size. > > > > My challenge, on the other side, is that I need to update the indexes > very > > frequently. So, do you think MMapDirectory is the solution? > > > > Thanks. > > > > On Mon, Jun 4, 2012 at 10:30 PM, Jack Krupansky <j...@basetechnology.com > >wrote: > > > >> From the javadoc for RAMDirectory: > >> > >> "Warning: This class is not intended to work with huge indexes. > Everything > >> beyond several hundred megabytes will waste resources (GC cycles), > because > >> it uses an internal buffer size of 1024 bytes, producing millions of > >> byte[1024] arrays. This class is optimized for small memory-resident > >> indexes. It also has bad concurrency on multithreaded environments. > >> > >> It is recommended to materialize large indexes on disk and use > >> MMapDirectory, which is a high-performance directory implementation > working > >> directly on the file system cache of the operating system, so copying > data > >> to Java heap space is not useful." > >> > >> -- Jack Krupansky > >> > >> -----Original Message----- From: Cheng > >> Sent: Monday, June 04, 2012 10:08 AM > >> To: java-user@lucene.apache.org > >> Subject: RAMDirectory unexpectedly slows > >> > >> > >> Hi, > >> > >> My apps need to read from and write to some big indexes frequently. So I > >> use RAMDirectory instead of FSDirectory, and give JVM about 2GB memory > >> size. > >> > >> I notice that the speed of reading and writing unexpectedly slows as the > >> size of the indexes increases. Since the usage of RAM is less than 20%, > I > >> think by default the RAMDirectory doesn't take advantage of the memory I > >> assigned to JVM. > >> > >> What are the steps to improve the reading and writing speed of > >> RAMDirectory? > >> > >> Thanks! > >> Jeff > >> > >> > ------------------------------**------------------------------**--------- > >> To unsubscribe, e-mail: java-user-unsubscribe@lucene.**apache.org< > java-user-unsubscr...@lucene.apache.org> > >> For additional commands, e-mail: java-user-help@lucene.apache.**org< > java-user-h...@lucene.apache.org> > >> > >> > > --------------------------------------------------------------------- > To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org > For additional commands, e-mail: java-user-h...@lucene.apache.org > >
