Re: Index size and performance degradation

2011-06-14 Thread Michael McCandless
Sorry, wrong email ;) Mike McCandless http://blog.mikemccandless.com On Tue, Jun 14, 2011 at 8:05 AM, Michael McCandless wrote: > Hmm, this sounds hairy :) > > Are you sure NRTCachingDir won't work for you? > > Mike McCandless > > http://blog.mikemccandless.com > > On Tue, Jun 14, 2011 at 5:58

Re: Index size and performance degradation

2011-06-14 Thread Michael McCandless
Hmm, this sounds hairy :) Are you sure NRTCachingDir won't work for you? Mike McCandless http://blog.mikemccandless.com On Tue, Jun 14, 2011 at 5:58 AM, Ganesh wrote: > Is it a bad idea to keep multiple shards in a single system? > > Regards > Ganesh > > - Original Message - > From: "T

Re: Index size and performance degradation

2011-06-14 Thread Ganesh
Is it a bad idea to keep multiple shards in a single system? Regards Ganesh - Original Message - From: "Toke Eskildsen" To: Sent: Tuesday, June 14, 2011 12:58 PM Subject: Re: Index size and performance degradation > On Sun, 2011-06-12 at 10:10 +0200, Itamar Syn-Hershko wrote: >> The

Re: Index size and performance degradation

2011-06-14 Thread Stefan Trcek
On Sunday 12 June 2011 22:12:01 Michael McCandless wrote: > Anyway, I don't think that's a good tradeoff, in general, for our > users, because very few apps truly require immediate consistency from > Lucene (can anyone give an example where their app depends on > immediate consistency...? For data

German compound decomposition (native speakers: help needed).

2011-06-14 Thread Dawid Weiss
First of all I should probably congratulate my fellow Germans -- Dirk Nowitzki's outstanding performance during this year's NBA finals will become part of the history of basketball. As a Pole, I admit I'm really freaking jealous. Now... back to the subject. A number of people have expressed an in

Re: Index size and performance degradation

2011-06-14 Thread mark harwood
Partitioning and replication are the keys to handling data and user volumes respectively. However, this approach introduces some other concerns over consistency and availability of content which I've tried to capture here: http://www.slideshare.net/MarkHarwood/patterns-for-large-scale-search Th

Re: Index size and performance degradation

2011-06-14 Thread Itamar Syn-Hershko
Thanks. Our product is pretty generic and we can't assume much on the hardware, as well as on usage. Some users would want low latency, others will prefer throughput. My job is to make as little compromise as possible... As for SSD, thats generally a good advice, except they seem to be faili

Re: Index size and performance degradation

2011-06-14 Thread Ganesh
We tried with more than 50 shards in the single system. Having multiple small index, indexes and optimizes the content faster. We use ParallelMultiSearcher to search across the index and the performance is really good. Now we plan to move to 64 Bit, so that we could use more RAM. Regards Ganesh

Re: Index size and performance degradation

2011-06-14 Thread Toke Eskildsen
On Sun, 2011-06-12 at 10:10 +0200, Itamar Syn-Hershko wrote: > The whole point of my question was to find out if and how to make > balancing on the SAME machine. Apparently thats not going to help and at > a certain point we will just have to prompt the user to buy more hardware... It really dep