Re: Are there any Lucene optimizations applicable to SSD?

2008-08-20 Thread Cedric Ho
> [Cedric: Yes] > >> However I can't figure out why some of these queries are slower. Some >> are complicated queries, yet others are just simple single term >> queries and doesn't seems to score lots of hits. There's no >> correlation between the number of terms or number of hits with the >> respo

Re: Are there any Lucene optimizations applicable to SSD?

2008-08-20 Thread Toke Eskildsen
On Wed, 2008-08-20 at 21:58 +0800, Cedric Ho wrote: Toke: > > Is it the same queries that are slow each time? [Cedric: Yes] > However I can't figure out why some of these queries are slower. Some > are complicated queries, yet others are just simple single term > queries and doesn't seems to scor

Re: Are there any Lucene optimizations applicable to SSD?

2008-08-20 Thread Cedric Ho
Hi Toke, >> Search response time. We used the search log from our production >> system and test it with SSD. The results shows that 75% of queries >> returns within 1 second, 90% returns in 2.5 seconds, the remaining 10% >> ranges from 2.5 seconds to less than 100 seconds. > > Are the sub-second r

Re: Are there any Lucene optimizations applicable to SSD?

2008-08-20 Thread Cedric Ho
Hi eks, On Wed, Aug 20, 2008 at 3:04 PM, eks dev <[EMAIL PROTECTED]> wrote: > The simplest sorting would be to sort your collection before indexing, > because Lucene will preserve order of added documents I think nutch sorts > index afterward somehow, but I do not know how this works The way

Re: Are there any Lucene optimizations applicable to SSD?

2008-08-20 Thread Toke Eskildsen
On Wed, 2008-08-20 at 00:25 +0800, Cedric Ho wrote: > Search response time. We used the search log from our production > system and test it with SSD. The results shows that 75% of queries > returns within 1 second, 90% returns in 2.5 seconds, the remaining 10% > ranges from 2.5 seconds to less than

Re: Are there any Lucene optimizations applicable to SSD?

2008-08-20 Thread eks dev
/jira/browse/LUCENE-1340 - Original Message > From: Cedric Ho <[EMAIL PROTECTED]> > To: java-user@lucene.apache.org > Sent: Wednesday, 20 August, 2008 3:28:36 AM > Subject: Re: Are there any Lucene optimizations applicable to SSD? > > Hi eks, > > My i

Re: Are there any Lucene optimizations applicable to SSD?

2008-08-19 Thread Cedric Ho
Hi eks, My index is fully optimized, but I wasn't aware that I can sort it by fields in Lucene. Could you elaborate on how to do that? By omitTf(), do you mean Fieldable.setOmitNorms(true)? I'll try that. Thanks, Cedric Ho > > if you have possibility to sort your index once in a while on somet

Re: Are there any Lucene optimizations applicable to SSD?

2008-08-19 Thread eks dev
hi Cedric, has nothing to do with SSD... but > > All queries involves a Date Range Filter and a Publication Filter. > We've used WrappingCachingFilters for the Publication Filter for there > are only a limited number of combinations for this filter. For the > Date Range Filter we just let it r

Re: Are there any Lucene optimizations applicable to SSD?

2008-08-19 Thread Cedric Ho
Hi, Thanks for the reply =) > > What aspect of performance do you find lacking? Is it searching or > indexing? While we've had stellar results for searches, indexing is just > so-so better than conventional harddisks. Search response time. We used the search log from our production system and te

Re: Are there any Lucene optimizations applicable to SSD?

2008-08-19 Thread Toke Eskildsen
On Tue, 2008-08-19 at 16:22 +0800, Cedric Ho wrote: [Lucene on SSD] > However it's still not good enough for our particular case. So I > wonder if there are any tips for optimizing lucene performance on > SSDs. What aspect of performance do you find lacking? Is it searching or indexing? While we

Are there any Lucene optimizations applicable to SSD?

2008-08-19 Thread Cedric Ho
Hi all, We are testing Lucene with SSD. No doubt the performance is much better than that of a normal hard disk. However it's still not good enough for our particular case. So I wonder if there are any tips for optimizing lucene performance on SSDs. For example, I saw that Lucene's BufferedIndex

Re: Lucene Optimizations

2007-04-03 Thread Otis Gospodnetic
Hi Nilesh, I have a few queries regarding optimizing lucene for search performance. 1. We index around 50 million text documents with index size greater than 40GB, and hence runtime performance is curcial. Our system has only simple keyword queries. Each search returns an object of type Hits whic

Lucene Optimizations

2007-04-03 Thread Nilesh Bansal
Hi all, I have a few queries regarding optimizing lucene for search performance. 1. We index around 50 million text documents with index size greater than 40GB, and hence runtime performance is curcial. Our system has only simple keyword queries. Each search returns an object of type Hits which