My data is categorized by date. About 14M+ docs per month, 37M+ terms. When I use 1G heap size to do search of 10 month index, I got OOM. The problem is I can't increase heap size in an easy way. I have several machines, all 32bit windows, 4G ram. And my goal is to index 10 year's data, plus more data every day ! If I put all of them together, I will need 8G+ ram to run search. Maybe another 8G+ ram to run indexwriter.
I think to split large index into smaller indexes and use a group of machines to work as one is more flexible and faster compare to one huge ram machine. Any suggestions ? beside more rams. 2009/11/16 Jacob Rhoden <jrho...@unimelb.edu.au>: > Not sure how large your index is, but it might be easier (if possible to > increase your memory) than to develop a fairly complicated alternative > strategy. > > On 16/11/2009, at 2:12 PM, Wenbo Zhao wrote: > >> Hi, all >> I'm facing a large index, on a x86 win platform which may not have big >> enough jvm heap space to hold the entire index. >> So, I think it's possible to split the index into several smaller >> indexes, run them in different jvm instances on different machine. >> Then for each query, I can concurrently run it one every indexes and >> merge the result together. >> This can be a workaround of OutOfMemory issue. >> But before I start to do this, I want to ask if Lucene already have a >> solution for things like this. >> Thanks. >> >> -- >> >> Best Regards, >> ZHAO, Wenbo >> >> ======================= >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org >> For additional commands, e-mail: java-user-h...@lucene.apache.org >> > > ____________________________________ > Information Technology Services, > The University of Melbourne > > Email: jrho...@unimelb.edu.au > Phone: +61 3 8344 2884 > Mobile: +61 4 1095 7575 > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org > For additional commands, e-mail: java-user-h...@lucene.apache.org > > -- Best Regards, ZHAO, Wenbo ======================= --------------------------------------------------------------------- To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org For additional commands, e-mail: java-user-h...@lucene.apache.org
