Sounds like you may need to have some sort of distributed system, I
just wanted to make sure you were aware of the cost/benifits of just
buying a big 62bit/8Gb ram machine, vs having to not only maintain and
power several 32 bit machines, but also maintain and support your now
more complicated code.
I have seen it too many times developers/companies spend so much money
in not just the initial development, but long term support and
maintenance that could have been simplified by just buying a bigger/
better more powerful machine in the first place.
I am interested to see what other people have to say about how to
solve your problem.
Best regards,
Jacob
On 16/11/2009, at 3:39 PM, Wenbo Zhao wrote:
My data is categorized by date. About 14M+ docs per month, 37M+
terms.
When I use 1G heap size to do search of 10 month index, I got OOM.
The problem is I can't increase heap size in an easy way.
I have several machines, all 32bit windows, 4G ram.
And my goal is to index 10 year's data, plus more data every day !
If I put all of them together, I will need 8G+ ram to run search.
Maybe another 8G+ ram to run indexwriter.
I think to split large index into smaller indexes and use a group of
machines to work as one is more flexible and faster compare to one
huge ram machine.
Any suggestions ? beside more rams.
2009/11/16 Jacob Rhoden <jrho...@unimelb.edu.au>:
Not sure how large your index is, but it might be easier (if
possible to
increase your memory) than to develop a fairly complicated
alternative
strategy.
On 16/11/2009, at 2:12 PM, Wenbo Zhao wrote:
Hi, all
I'm facing a large index, on a x86 win platform which may not have
big
enough jvm heap space to hold the entire index.
So, I think it's possible to split the index into several smaller
indexes, run them in different jvm instances on different machine.
Then for each query, I can concurrently run it one every indexes and
merge the result together.
This can be a workaround of OutOfMemory issue.
But before I start to do this, I want to ask if Lucene already
have a
solution for things like this.
Thanks.
--
Best Regards,
ZHAO, Wenbo
=======================
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-user-h...@lucene.apache.org
____________________________________
Information Technology Services,
The University of Melbourne
Email: jrho...@unimelb.edu.au
Phone: +61 3 8344 2884
Mobile: +61 4 1095 7575
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-user-h...@lucene.apache.org
--
Best Regards,
ZHAO, Wenbo
=======================
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-user-h...@lucene.apache.org
____________________________________
Information Technology Services,
The University of Melbourne
Email: jrho...@unimelb.edu.au
Phone: +61 3 8344 2884
Mobile: +61 4 1095 7575
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-user-h...@lucene.apache.org
