Hello Anshum,
My index is growing 1 million documents per day. Initially i planned to have
a single database but the sorting of one or more fields consumes more RAM.
Whether sharding the index would also consume the same.
My application should co-exist with other application of my product and my
app could get 1 GB of RAM. Search speed is fine but i need to display the
result in the sorted order.
I thought to keep 7 days of documents in one index and create one more after
the 7 days. After 30 days the first index may get deleted. I need to keep
the documents in the index DB for 30 days. My Index DB is in HDD.
I want to the pros and cons of sharding. I think maintance of the DB becomes
easier.
It would be very much helpful, if you share some of your thoughts.
Regards
Ganesh
----- Original Message -----
From: "Anshum" <[EMAIL PROTECTED]>
To: <java-user@lucene.apache.org>
Sent: Friday, October 03, 2008 9:48 PM
Subject: Re: Single searcher vs Multi Searcher
Hi Ganesh,
I have experimented with sharded indexes and they seem to benefit
me(atleast
in my case). I would like to know a few things before I answer your
question:
1. Do you have a reasonable criteria ( a calculated one) to shard the
indexes?
2. How do you plan to split the index? Is it going to be document based
(which I guess it should be as otherwise you would have to build a
complete
distributed system)
3. Do you plan to put your indexes on the RAM or on (physically) seperate
HDDs?
Though all said and done, sharded indexes are a good approach, if done the
right way.
--
Anshum Gupta
Naukri Labs!
http://ai-cafe.blogspot.com
The facts expressed here belong to everybody, the opinions to me. The
distinction is yours to draw............
On Fri, Oct 3, 2008 at 3:01 PM, Ganesh <[EMAIL PROTECTED]> wrote:
Hello all,
My indexing is growing by 1 million records per day and the memory
consumption of the searcher object is quite high.
There are different opinion in the groups. Few suggest to use single
database and few to use sharding. My Database has 10 million records now
and
it might go till 30 million or more. I plan to shard the index. but
Multisearcher will give me benifit.
Regards
Ganesh
Send instant messages to your online friends
http://in.messenger.yahoo.com
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
Send instant messages to your online friends http://in.messenger.yahoo.com
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
