Hi Tim,
is it possible that the slow queries contains terms that are very
common in your index? If so you could replace those clauses with a
filter. This would impact the score as filters does nothing with that,
but if your query contains enough other clauses that should not be a
problem.
That is how I've in several applications managed to solve the problem
you describe.
karl
4 dec 2008 kl. 21.27 skrev Tim Sturge:
Hi all,
I have an interesting problem with my query traffic. Most of the
queries run
in a fairly short amount of time (< 100ms) but a few take over
1000ms. These
queries are predominantly those with a huge number of hits (>1
million hits
in a >100 million document index). The time taken (as far as I can
tell) is
for lucene to sit there while it scores and sorts all these results.
However it turns out these queries really don’t have top results.
That is,
of the million documents, there are easily 10000 which are decent
results
(basically those above some threshold score). Frankly, just
returning some
consistent (so paging and reload work) but
otherwise arbitrary ranking of these 10000 results would be more
than good
enough.
It seems to me that a solution would be to impose some sort of
pseudo-random
filter (e.g. consider only every n-th document assuming they are
uniformly
distributed). I’m wondering if anyone else has experience with this
sort of
issue and what solutions they have found to work well in practice.
Thanks,
Tim
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
