Re: Lucene Index size v/s available memory

2009-12-23 Thread Erick Erickson
The size of your index isn't a very useful number without knowing a significant amount about the structure of your index. Depending upon what's stored, what's indexed and what kind of searching you're doing (e.g. sorting?) it varies. About all we can say is that you'll probably need less than 100G.

Re: Lucene Index size v/s available memory

2009-12-23 Thread Ian Lea
Hi 24Gb RAM for a 100Gb index is likely to be plenty. You don't have a huge amount of control over what lucene loads in memory, but take a look at termInfosIndexDivisor in IndexReader. And I believe that omitting field norms (Field.setOmitNorms) may help too. Googling for "lucene memory usage"

Lucene Index size v/s available memory

2009-12-22 Thread Shakti Purohit
We are required to find out how much percentage/part of lucene index needs to be in memory for acceptable search response time. The index size we have is around 100 GB while the available memory is 24 GB. Since we do not have the option of loading whole of the index in memory we wanted to know w