Re: RAMDirectory unexpectedly slows

2012-06-30 Thread Michael McCandless
t; >> This is managed by your operating system. In general OS kernels like >> Linux >> >> or Windows use all free memory to cache disk accesses. >> >> >> >> - >> >> Uwe Schindler >> >> H.-H.-Meier-Allee 63, D-28213 Bremen &

Re: RAMDirectory unexpectedly slows

2012-06-30 Thread Cheng
like > Linux > >> or Windows use all free memory to cache disk accesses. > >> > >> - > >> Uwe Schindler > >> H.-H.-Meier-Allee 63, D-28213 Bremen > >> http://www.thetaphi.de > >> eMail: u...@thetaphi.de > >> > >

Re: RAMDirectory unexpectedly slows

2012-06-18 Thread Cheng
;> H.-H.-Meier-Allee 63, D-28213 Bremen > >> http://www.thetaphi.de > >> eMail: u...@thetaphi.de > >> > >> > >> > -Original Message- > >> > From: Cheng [mailto:zhoucheng2...@gmail.com] > >> > Sent: Monda

Re: RAMDirectory unexpectedly slows

2012-06-18 Thread Michael McCandless
ee memory to cache disk accesses. >> >> - >> Uwe Schindler >> H.-H.-Meier-Allee 63, D-28213 Bremen >> http://www.thetaphi.de >> eMail: u...@thetaphi.de >> >> >> > -Original Message- >> > From: Cheng [mailto:zhouc

Re: RAMDirectory unexpectedly slows

2012-06-16 Thread Cheng
hetaphi.de > eMail: u...@thetaphi.de > > > > -Original Message- > > From: Cheng [mailto:zhoucheng2...@gmail.com] > > Sent: Monday, June 04, 2012 6:10 PM > > To: java-user@lucene.apache.org > > Subject: Re: RAMDirectory unexpectedly slows > >

RE: RAMDirectory unexpectedly slows

2012-06-04 Thread Uwe Schindler
cheng2...@gmail.com] > Sent: Monday, June 04, 2012 6:10 PM > To: java-user@lucene.apache.org > Subject: Re: RAMDirectory unexpectedly slows > > Can I control the size of ram given to either MMapDirectory or > ByteBufferDirectory? > > On Mon, Jun 4, 2012 at 11:42 PM, U

Re: RAMDirectory unexpectedly slows

2012-06-04 Thread Cheng
Can I control the size of ram given to either MMapDirectory or ByteBufferDirectory? On Mon, Jun 4, 2012 at 11:42 PM, Uwe Schindler wrote: > Hi, > > If you are using MMapDirectory or this ByteBufferDirectory (which is > similar to the first) the used RAM is outside JVM heap, it is in the FS > cac

Re: RAMDirectory unexpectedly slows

2012-06-04 Thread Uwe Schindler
Hi, If you are using MMapDirectory or this ByteBufferDirectory (which is similar to the first) the used RAM is outside JVM heap, it is in the FS cache of the OS kernel. Giving too much memory to the JVM penalizes the OS cache, so give only as much as the App needs. Lucene and the OS kernel will

Re: RAMDirectory unexpectedly slows

2012-06-04 Thread Jason Rutherglen
> What about the ByteBufferDirectory? Can this specific directory utilize the > 2GB memory I grant to the app? BBD places the byte objects outside of the heap, so increasing the heap size is only going to rob the system IO cache. With Lucene the heap is only used for field caches and the terms di

Re: RAMDirectory unexpectedly slows

2012-06-04 Thread Cheng
Please shed more insight into the difference between JVM heap size and the memory size used by Lucene. What I am getting at is that no matter however much ram I give my apps, Lucene can't utilize it. Is that right? What about the ByteBufferDirectory? Can this specific directory utilize the 2GB me

Re: RAMDirectory unexpectedly slows

2012-06-04 Thread Jason Rutherglen
If you want the index to be stored completely in RAM, there is the ByteBuffer directory [1]. Though I do not see the point in putting an index in RAM, it will be cached in RAM regardless in the OS system IO cache. 1. https://github.com/elasticsearch/elasticsearch/blob/master/src/main/java/org/ap

Re: RAMDirectory unexpectedly slows

2012-06-04 Thread Cheng
My indexes are 500MB+. So it seems like that RAMDirectory is not good for that big a size. My challenge, on the other side, is that I need to update the indexes very frequently. So, do you think MMapDirectory is the solution? Thanks. On Mon, Jun 4, 2012 at 10:30 PM, Jack Krupansky wrote: > Fro

Re: RAMDirectory unexpectedly slows

2012-06-04 Thread Jack Krupansky
From the javadoc for RAMDirectory: "Warning: This class is not intended to work with huge indexes. Everything beyond several hundred megabytes will waste resources (GC cycles), because it uses an internal buffer size of 1024 bytes, producing millions of byte[1024] arrays. This class is optimi