Re: How to use concurrency efficiently

2013-04-03 Thread Paul Bell
All, Sorry, but I inadvertenly put my post re MultiFieldQueryParser in the wrong thread (wrong subject via cut and paste). Igor, thank you for the reply. I will look into what you suggest. -Paul On Wed, Apr 3, 2013 at 6:58 AM, Igor Shalyminov wrote: > I personally use SpanNearQuey (span posit

Re: How to use concurrency efficiently

2013-04-03 Thread Igor Shalyminov
I personally use SpanNearQuey (span positions are always needed), and for different fields I use FieldMaskingSpanQuery class. I just choose one field name and then mask each SpanTermQuery's real field name with this field via wrapper. Maybe it can help. -- Igor 03.04.2013, 06:59, "Paul" : > H

Re: How to use concurrency efficiently

2013-04-02 Thread Paul
Hi, I've experimented a bit with MultiFieldQueryParser (http://lucene.apache.org/core/4_2_0/queryparser/org/apache/lucene/queryparser/classic/MultiFieldQueryParser.html) But it seems to search for each of a query's terms in each field specified in the constructor. So, as the doc says, if you q

RE: How to use concurrency efficiently

2013-04-02 Thread Uwe Schindler
> -Original Message- > From: Igor Shalyminov [mailto:ishalymi...@yandex-team.ru] > Sent: Tuesday, April 02, 2013 9:58 PM > To: java-user@lucene.apache.org > Subject: Re: How to use concurrency efficiently > > These are not document hits but text hits (to be more specific, spa

Re: How to use concurrency efficiently

2013-04-02 Thread Igor Shalyminov
These are not document hits but text hits (to be more specific, spans). For the search result it is necessary to have the precise number of document and text hits and a relatively small number of matched text snippets. I've tried several approaches to optimize the search algorithm but they didn't

Re: How to use concurrency efficiently

2013-04-02 Thread Adrien Grand
On Tue, Apr 2, 2013 at 4:39 PM, Igor Shalyminov wrote: > Yes, the number of documents is not too large (about 90 000), but the queries > are very hard. Although they're just boolean, a typical query can produce a > result with tens of millions of hits. How can there be tens of millions of hits

Re: How to use concurrency efficiently

2013-04-02 Thread Igor Shalyminov
Yes, the number of documents is not too large (about 90 000), but the queries are very hard. Although they're just boolean, a typical query can produce a result with tens of millions of hits. Single-threadedly such a query runs ~20 seconds, which is too slow. therefore, multithreading is vital f

Re: How to use concurrency efficiently

2013-04-02 Thread Adrien Grand
On Tue, Apr 2, 2013 at 2:29 PM, Igor Shalyminov wrote: > Hello! Hi Igor, > I have a ~20GB index and try to make a concurrent search over it. > The index has 16 segments, I run SpanQuery.getSpans() on each segment > concurrently. > I see really small performance improvement of searching concurre

How to use concurrency efficiently

2013-04-02 Thread Igor Shalyminov
Hello! I have a ~20GB index and try to make a concurrent search over it. The index has 16 segments, I run SpanQuery.getSpans() on each segment concurrently. I see really small performance improvement of searching concurrently. I suppose, the reason is that the sizes of the segments are very non-