Hello,
We've created a log management system using lucene 4.3. Each log has about 10 fields and all of them are stored. We store each hour of the logs in a separate folder so when someone runs a query only the folders specified in the time frame are searched. The indexes are loaded using the mmap directory and the search is only done on the required folders. The problem we have is that when we run a query on three weeks of data which has around 250M documents it takes around 200 seconds. Which is too much. We're using servers running virtualization and the storage is on SAN. Is there anyway we can increase our performance? Can I distribute the indexes on different servers and distribute the queries on them and aggregate the results? By the way we have sorting on the queries. Has the splitting the indexes in different folders created a problem? Is this performance bad in the first place? Can I be expecting it to increase? Sorry for the different questions. Thanks for your help. Hamed
