[
https://issues.apache.org/jira/browse/SOLR-17642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Varun Thacker updated SOLR-17642:
---------------------------------
Description:
First reported on
[https://lists.apache.org/thread/xn1xg5szxhrs0sbcy4gmx4cvohy6flvh|https://lists.apache.org/thread/xn1xg5szxhrs0sbcy4gmx4cvohy6flvh,],
Dr. Andreas Moll reported a failing query.
I was able to reproduce is consistently locally on 9.7 and 9.8 using Dr.
Andreas Moll's repro steps.
There are two bugs in play here
*The Solr side bug:*
* Solr 9.7 added SOLR-13350 but disabled multi-threaded search by default. I
can confirm the failing query follows this code path
{code:java}
if (!MultiThreadedSearcher.allowMT(pf.postFilter, cmd)) {
log.trace("SINGLE THREADED search, skipping collector manager in
getDocListNC");{code}
* However when following that code path the query goes to
AbstractKnnVectorQuery and this is where executor thread pool gets used
{code:java}
TimeLimitingKnnCollectorManager knnCollectorManager =
new TimeLimitingKnnCollectorManager(
getKnnCollectorManager(k, indexSearcher), indexSearcher.getTimeout());
TaskExecutor taskExecutor = indexSearcher.getTaskExecutor();
{code}
* This executor was added in SOLR-13350 in CoreContainer(
indexSearcherExecutor) . If you follow the SolrIndexSearcher code that calls
{code:java}
In SolrIndexSearcher.java
super(wrapReader(core, r),
core.getCoreContainer().getIndexSearcherExecutor());{code}
{code:java}
In IndexSearcher.java
this.executor = executor;
this.taskExecutor =
executor == null ? new TaskExecutor(Runnable::run) : new
TaskExecutor(executor);{code}
I think the idea in SOLR-13350 was create an indexSearcherExector, but only use
it when multiThreaded=true as a query param (default=false). The fact we
initialised indexSearcherExector means all KNN queries use that thread pool and
there is no way to disable that.
*Now the Lucene Bug:*
[~benwtrent] noted on
[https://lists.apache.org/thread/ftm6f32f2nsoys6noxx8p2ygpwnlfjc9] that
Multi-threaded vector search over multiple segments can lead to inconsistent
results
So the fact Solr by default always uses multi threaded search 9.7+ triggers the
lucene bug. But I think in Solr we should also have a way to not use
multi-threaded KNN like we do for other queries.
was:
First reported on
[https://lists.apache.org/thread/xn1xg5szxhrs0sbcy4gmx4cvohy6flvh|https://lists.apache.org/thread/xn1xg5szxhrs0sbcy4gmx4cvohy6flvh,],
Dr. Andreas Moll reported a failing query.
I was able to reproduce is consistently locally on 9.7 and 9.8 using Dr.
Andreas Moll's repro steps.
There are two bugs in play here
*The Solr side bug:*
* Solr 9.7 added https://issues.apache.org/jira/browse/SOLR-13350 but disabled
multi-threaded search by default. I can confirm the failing query follows this
code path
{code:java}
if (!MultiThreadedSearcher.allowMT(pf.postFilter, cmd)) {
log.trace("SINGLE THREADED search, skipping collector manager in
getDocListNC");{code}
* However when following that code path the query goes to
AbstractKnnVectorQuery and this is where executor thread pool gets used
{code:java}
TimeLimitingKnnCollectorManager knnCollectorManager =
new TimeLimitingKnnCollectorManager(
getKnnCollectorManager(k, indexSearcher), indexSearcher.getTimeout());
TaskExecutor taskExecutor = indexSearcher.getTaskExecutor();
{code}
* This executor was added in SOLR-13350 in CoreContainer(
indexSearcherExecutor) . If you follow the SolrIndexSearcher code that calls
{code:java}
In SolrIndexSearcher.java
super(wrapReader(core, r),
core.getCoreContainer().getIndexSearcherExecutor());{code}
{code:java}
In IndexSearcher.java
this.executor = executor;
this.taskExecutor =
executor == null ? new TaskExecutor(Runnable::run) : new
TaskExecutor(executor);{code}
I think the idea in SOLR-13350 was create an indexSearcherExector, but only use
it when multiThreaded=true as a query param (default=false). The fact we
initialised indexSearcherExector means all KNN queries use that thread pool and
there is no way to disable that.
*Now the Lucene Bug:*
[~benwtrent] noted on
[https://lists.apache.org/thread/ftm6f32f2nsoys6noxx8p2ygpwnlfjc9] that
Multi-threaded vector search over multiple segments can lead to inconsistent
results
So the fact Solr by default always uses multi threaded search 9.7+ triggers the
lucene bug. But I think in Solr we should also have a way to not use
multi-threaded KNN like we do for other queries.
> Knn queries use multi threaded search unintentionally
> -------------------------------------------------------
>
> Key: SOLR-17642
> URL: https://issues.apache.org/jira/browse/SOLR-17642
> Project: Solr
> Issue Type: Bug
> Reporter: Varun Thacker
> Priority: Major
>
> First reported on
> [https://lists.apache.org/thread/xn1xg5szxhrs0sbcy4gmx4cvohy6flvh|https://lists.apache.org/thread/xn1xg5szxhrs0sbcy4gmx4cvohy6flvh,],
> Dr. Andreas Moll reported a failing query.
> I was able to reproduce is consistently locally on 9.7 and 9.8 using Dr.
> Andreas Moll's repro steps.
> There are two bugs in play here
> *The Solr side bug:*
> * Solr 9.7 added SOLR-13350 but disabled multi-threaded search by default. I
> can confirm the failing query follows this code path
>
> {code:java}
> if (!MultiThreadedSearcher.allowMT(pf.postFilter, cmd)) {
> log.trace("SINGLE THREADED search, skipping collector manager in
> getDocListNC");{code}
>
> * However when following that code path the query goes to
> AbstractKnnVectorQuery and this is where executor thread pool gets used
> {code:java}
> TimeLimitingKnnCollectorManager knnCollectorManager =
> new TimeLimitingKnnCollectorManager(
> getKnnCollectorManager(k, indexSearcher), indexSearcher.getTimeout());
> TaskExecutor taskExecutor = indexSearcher.getTaskExecutor();
> {code}
> * This executor was added in SOLR-13350 in CoreContainer(
> indexSearcherExecutor) . If you follow the SolrIndexSearcher code that calls
> {code:java}
> In SolrIndexSearcher.java
> super(wrapReader(core, r),
> core.getCoreContainer().getIndexSearcherExecutor());{code}
> {code:java}
> In IndexSearcher.java
> this.executor = executor;
> this.taskExecutor =
> executor == null ? new TaskExecutor(Runnable::run) : new
> TaskExecutor(executor);{code}
> I think the idea in SOLR-13350 was create an indexSearcherExector, but only
> use it when multiThreaded=true as a query param (default=false). The fact we
> initialised indexSearcherExector means all KNN queries use that thread pool
> and there is no way to disable that.
>
> *Now the Lucene Bug:*
> [~benwtrent] noted on
> [https://lists.apache.org/thread/ftm6f32f2nsoys6noxx8p2ygpwnlfjc9] that
> Multi-threaded vector search over multiple segments can lead to inconsistent
> results
>
> So the fact Solr by default always uses multi threaded search 9.7+ triggers
> the lucene bug. But I think in Solr we should also have a way to not use
> multi-threaded KNN like we do for other queries.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org
