On Nov 13, 2005, at 8:19 PM, Friedland, Zachary (EDS - Strategy) wrote:
What is the largest lucene index that has been built? We're looking to build a sort of data warehouse that will hold transaction log files as long as possible. This index would grow at the rate of 10 million documents per month indefinitely. Is there a limit where lucene will fail? What should we expect to happen as this index grows?
Our index has about 3 million documents and is about 11 Gig in size. It has become unwieldy to manage at that size so I have been working to reduce the size of our Stored data and it looks like we're going to get the index size down below 2 Gig. The overall performance directly seems to correlate the to the size of the index. Try to keep your Stored fields minimal as it would appear that the number of documents is far less an issue than the physical size of the index file. At some point, we will probably start to split our index if it gets above 5 Gig again. However, my initial tests with the MultiReader have been disappointing. Hopefully, a faster MultiReader will be available by the time we cross that bridge.
Charles. --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
