ping. Sorry for the long email but I prefer to provide all information first.
On Mon, May 12, 2008 at 12:13 PM, Stephane Nicoll <[EMAIL PROTECTED]> wrote: > I tried all this and I am confused about the result. I am trying to > implement an hybrid query handler where I fetch the IDs from a > database criteria and the IDs from a full text lucene query and I > intersect them to return the result to the user. The database query > and the intersection works fine even with high load. However the > lucene query is much slower when the number of concurrent users > raises. > > Here is what I am doing on the lucene side > > final QueryParser queryParser = new > QueryParser(criteria.getDefaultField(), analyzer); > final Query q = queryParser.parse(criteria.getFullTextQuery()); > // Index Searcher is shared for all threads and is not > reopened during the load test > final IndexSearcher indexSearcher = getIndexSearcher(); > final Set<Long> result = new TreeSet<Long>(); > indexSearcher.search(q, new HitCollector() { > public void collect(int i, float v) { > try { > final Document d = > indexSearcher.getIndexReader().document(i, new FieldSelector() { > public FieldSelectorResult accept(String s) { > if (s.equals(CatalogItem.ATTR_ID)) { > return FieldSelectorResult.LOAD; > } else { > return FieldSelectorResult.NO_LOAD; > } > } > }); > result.add(Long.parseLong(d.get(CatalogItem.ATTR_ID))); > } catch (IOException e) { > throw new RuntimeException("Could not collect > lucene IDs", e); > } > } > }); > return result; > > > When running with one thread, I have the following figures per test: > > Database query is done in[125 msecs] (size=598] > Lucene query is done in[80 msecs (size=15204] > Intersect is done in[4 msecs] (size=103] > Hybrid query is done in[97 msecs] > > -> 327 msec / user > > When running with ten threads, I have the following figures per user per > test: > > Database query is done in[222 msecs] (size=94] > Lucene query is done in[2364 msecs (size=15367] > Intersect is done in[0 msecs] (size=12] > Hybrid query is done in[18 msecs] > > -> 2.5 sec / user !! > > I am just wondering how I can improve this. Clearly there is something > wrong in my code since it's much slower with multiple threads running > concurrently on the same index. The size of the index is 5Mb, I only > store: > > * an "id" field (which is the primary key of the related object in the db > * a "class" field which is the class nazme of the related object > (Hibernate search does that for me) > > The "keywords" field is indexed but not stored as it is a > representation of other data stored in the db. The searches are > performed on the keywords field only ("foo AND bar" is a typical > query) > > Any help is appreciated. If you also know a Spring bean that could > take care of opening/closing the index readers properly, let me know. > Hibernate Search introduces deadlock with multiple threads and the > lucene integration in spring modules does not seeem to do what I want. > > Thanks, > Stéphane > > > > > On Sat, May 10, 2008 at 8:05 PM, Patrick Turcotte <[EMAIL PROTECTED]> wrote: > > Did you try the IndexSearcher.doc(int i, FieldSelector fieldSelector) > method? > > > > Could be faster because Lucene don't have do "prepare" the whole document. > > > > Patrick > > > > > > On Sat, May 10, 2008 at 9:35 AM, Stephane Nicoll > > <[EMAIL PROTECTED]> wrote: > > > > > > > From the FAQ: > > > > > > "Don't iterate over more hits than needed. > > > Iterating over all hits is slow for two reasons. Firstly, the search() > > > method that returns a Hits object re-executes the search internally > > > when you need more than 100 hits. Solution: use the search method that > > > takes a HitCollector instead." > > > > > > I had a look to HitCollector but it returns the documentId and the > > > javadoc recommends not fetching the original query there. > > > > > > I have to return *one* indexed field from the query result and > > > currently I am iterating on all results and it's slow. Can you explain > > > a bit more how I could improve this? > > > > > > Thanks, > > > Stéphane > > > > > > > > > -- > > > Large Systems Suck: This rule is 100% transitive. If you build one, > > > you suck" -- S.Yegge > > > > > > > > --------------------------------------------------------------------- > > > To unsubscribe, e-mail: [EMAIL PROTECTED] > > > > > For additional commands, e-mail: [EMAIL PROTECTED] > > > > > > > > > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: [EMAIL PROTECTED] > > > > > > For additional commands, e-mail: [EMAIL PROTECTED] > > > > > > > > -- > > > Large Systems Suck: This rule is 100% transitive. If you build one, > you suck" -- S.Yegge > -- Large Systems Suck: This rule is 100% transitive. If you build one, you suck" -- S.Yegge --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]