This is managed by your operating system. In general OS kernels like Linux or Windows use all free memory to cache disk accesses.
----- Uwe Schindler H.-H.-Meier-Allee 63, D-28213 Bremen http://www.thetaphi.de eMail: u...@thetaphi.de > -----Original Message----- > From: Cheng [mailto:zhoucheng2...@gmail.com] > Sent: Monday, June 04, 2012 6:10 PM > To: java-user@lucene.apache.org > Subject: Re: RAMDirectory unexpectedly slows > > Can I control the size of ram given to either MMapDirectory or > ByteBufferDirectory? > > On Mon, Jun 4, 2012 at 11:42 PM, Uwe Schindler <u...@thetaphi.de> wrote: > > > Hi, > > > > If you are using MMapDirectory or this ByteBufferDirectory (which is > > similar to the first) the used RAM is outside JVM heap, it is in the > > FS cache of the OS kernel. Giving too much memory to the JVM penalizes > > the OS cache, so give only as much as the App needs. Lucene and the OS > > kernel will then utilize the remaining memory for caching. > > > > Please read docs of MMapDirectory and inform yourself about mmap in e.g. > > Wikipedia. > > > > Uwe > > -- > > Uwe Schindler > > H.-H.-Meier-Allee 63, 28213 Bremen > > http://www.thetaphi.de > > > > > > > > Cheng <zhoucheng2...@gmail.com> schrieb: > > > > Please shed more insight into the difference between JVM heap size and > > the memory size used by Lucene. > > > > What I am getting at is that no matter however much ram I give my > > apps, Lucene can't utilize it. Is that right? > > > > What about the ByteBufferDirectory? Can this specific directory > > utilize the 2GB memory I grant to the app? > > > > On Mon, Jun 4, 2012 at 10:58 PM, Jason Rutherglen < > > jason.rutherg...@gmail.com> wrote: > > > > > If you want the index to be stored completely in RAM, there is the > > > ByteBuffer directory [1]. Though I do not see the point in putting > > > an index in RAM, it will be cached in RAM regardless in the OS > > > system IO cache. > > > > > > 1. > > > > > https://github.com/elasticsearch/elasticsearch/blob/master/src/main/ja > > va/org/apache/lucene/store/bytebuffer/ByteBufferDirectory.java > > > > > > On Mon, Jun 4, 2012 at 10:55 AM, Cheng <zhoucheng2...@gmail.com> > wrote: > > > > My indexes are 500MB+. So it seems like that RAMDirectory is not > > > > good > > for > > > > that big a size. > > > > > > > > My challenge, on the other side, is that I need to update the > > > > indexes > > > very > > > > frequently. So, do you think MMapDirectory is the solution? > > > > > > > > Thanks. > > > > > > > > On Mon, Jun 4, 2012 at 10:30 PM, Jack Krupansky < > > j...@basetechnology.com > > > >wrote: > > > > > > > >> From the javadoc for RAMDirectory: > > > >> > > > >> "Warning: This class is not intended to work with huge indexes. > > > Everything > > > >> beyond several hundred megabytes will waste resources (GC > > > >> cycles), > > > because > > > >> it uses an internal buffer size of 1024 bytes, producing millions > > > >> of byte[1024] arrays. This class is optimized for small > > > >> memory-resident indexes. It also has bad concurrency on multithreaded > environments. > > > >> > > > >> It is recommended to materialize large indexes on disk and use > > > >> MMapDirectory, which is a high-performance directory > > > >> implementation > > > working > > > >> directly on the file system cache of the operating system, so > > > >> copying > > > data > > > >> to Java heap space is not useful." > > > >> > > > >> -- Jack Krupansky > > > >> > > > >> -----Original Message----- From: Cheng > > > >> Sent: Monday, June 04, 2012 10:08 AM > > > >> To: java-user@lucene.apache.org > > > >> Subject: RAMDirectory unexpectedly slows > > > >> > > > >> > > > >> Hi, > > > >> > > > >> My apps need to read from and write to some big indexes frequently. > > So I > > > >> use RAMDirectory instead of FSDirectory, and give JVM about 2GB > > > >> memory size. > > > >> > > > >> I notice that the speed of reading and writing unexpectedly slows > > > >> as > > the > > > >> size of the indexes increases. Since the usage of RAM is less > > > >> than > > 20%, > > > I > > > >> think by default the RAMDirectory doesn't take advantage of the > > memory I > > > >> assigned to JVM. > > > >> > > > >> What are the steps to improve the reading and writing speed of > > > >> RAMDirectory? > > > >> > > > >> Thanks! > > > >> Jeff > > > >> > > > >> > > >_____________________________________________ > > **_____________________________________________ > > **--------- > > > >> To unsubscribe, e-mail: > > > >> java-user-unsubscribe@lucene.**apache.org< > > > java-user-unsubscr...@lucene.apache.org> > > > >> For additional commands, e-mail: > > > >> java-user-help@lucene.apache.**org< > > > java-user-h...@lucene.apache.org> > > > >> > > > >> > > > > > >_____________________________________________ > > > > > To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org > > > For additional commands, e-mail: java-user-h...@lucene.apache.org > > > > > > > > > > --------------------------------------------------------------------- To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org For additional commands, e-mail: java-user-h...@lucene.apache.org
