Re: Poor QPS with highlighting

2009-02-05 Thread Jason Rutherglen
http://en.wikipedia.org/wiki/Google_platform Document server summarization On Thu, Feb 5, 2009 at 12:57 PM, Michael Stoppelman wrote: > On Thu, Feb 5, 2009 at 12:47 PM, Michael Stoppelman >wrote: > > > > > > > On Thu, Feb 5, 2009 at 9:05 AM, Jason Rutherglen < > > jason.rutherg...@gmail.com> wr

Re: Poor QPS with highlighting

2009-02-05 Thread Michael Stoppelman
On Thu, Feb 5, 2009 at 12:47 PM, Michael Stoppelman wrote: > > > On Thu, Feb 5, 2009 at 9:05 AM, Jason Rutherglen < > jason.rutherg...@gmail.com> wrote: > >> Google uses dedicated highlighting servers. Maybe this architecture would >> work for you. >> > > What's your reference? I used to work at

Re: Poor QPS with highlighting

2009-02-05 Thread Michael Stoppelman
On Thu, Feb 5, 2009 at 9:05 AM, Jason Rutherglen wrote: > Google uses dedicated highlighting servers. Maybe this architecture would > work for you. > What's your reference? I used to work at Google. > > On Mon, Feb 2, 2009 at 11:24 PM, Michael Stoppelman >wrote: > > > Hi all, > > > > My sear

Re: Poor QPS with highlighting

2009-02-05 Thread Jason Rutherglen
Google uses dedicated highlighting servers. Maybe this architecture would work for you. On Mon, Feb 2, 2009 at 11:24 PM, Michael Stoppelman wrote: > Hi all, > > My search backends are only able to eek out 13-15 qps even with the entire > index in memory (this makes it very expensive to scale). A

RE: Poor QPS with highlighting

2009-02-05 Thread Beard, Brian
009 3:53 PM To: java-user@lucene.apache.org Subject: Re: Poor QPS with highlighting > Can you describe this in a little more detail; I'm not exactly sure what you > mean. > Break your large text documents into multiple Lucene documents. Rather than dividing them up into entirely di

Re: Poor QPS with highlighting

2009-02-04 Thread Michael Stoppelman
Thanks Mark for the explanation. I think your solution would definitely change the tf-idf scoring for documents since your field is now split up over multiple docs. One option to get around the changing scoring would be to to run a completely separate index for highlighting (with the overlapping d

Re: Poor QPS with highlighting

2009-02-03 Thread markharw00d
Can you describe this in a little more detail; I'm not exactly sure what you mean. Break your large text documents into multiple Lucene documents. Rather than dividing them up into entirely discreet chunks of text consider storing/indexing *overlapping* sections of text with an overlap as

Re: Poor QPS with highlighting

2009-02-03 Thread Michael Stoppelman
On Tue, Feb 3, 2009 at 1:14 AM, mark harwood wrote: > >>My documents are quite big sometimes up to 300ktokens. > > You could look at indexing them as seperate documents using overlapping > sections of text. Erik used this for one of his projects. > Can you describe this in a little more detail; I

Re: Poor QPS with highlighting

2009-02-03 Thread mark harwood
>>My documents are quite big sometimes up to 300ktokens. You could look at indexing them as seperate documents using overlapping sections of text. Erik used this for one of his projects. Cheers Mark - Original Message From: Michael Stoppelman To: java-user@lucene.apache.org Sent: Tu