Eric Pugh created SOLR-18348:
--------------------------------
Summary: `buildOnCommit=true` suggester rebuild blocks commit (and
the client request) indefinitely with no timeout
Key: SOLR-18348
URL: https://issues.apache.org/jira/browse/SOLR-18348
Project: Solr
Issue Type: Bug
Components: Suggester
Affects Versions: 9.10.1
Reporter: Eric Pugh
Assignee: Eric Pugh
h3. Problem
When a suggester is configured with `buildOnCommit=true`, a slow suggester
rebuild does not just make that commit take longer in the background — it
synchronously blocks the commit call itself, and therefore the client/update
request waiting on it, with *no timeout*. Under sustained heavy indexing
combined with throttled storage (e.g. an Azure disk after burst credits are
exhausted), this manifests as outright request timeouts requiring manual
operator intervention, rather than graceful degradation.
A customer hit this in production: ~4 million product documents being indexed
against a suggester with an I/O-intensive `AnalyzingInfixSuggester`
configuration, on a disk that had exhausted its burst-credit allowance.
Indexing requests began timing out. Thread dumps showed threads stuck in:
{code}
"searcherExecutor-250-thread-3-processing-...-core_node4" ... RUNNABLE
at org.apache.lucene.util.compress.LZ4.decompress(LZ4.java:138)
at
org.apache.lucene.codecs.lucene90.LZ4WithPresetDictCompressionMode$LZ4WithPresetDictDecompressor.decompress(...)
at
org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsReader...
at
org.apache.lucene.search.suggest.DocumentDictionary$DocumentInputIterator.next(DocumentDictionary.java:169)
at
org.apache.lucene.search.suggest.analyzing.AnalyzingInfixSuggester.build(AnalyzingInfixSuggester.java:346)
{code}
h3. Root cause
* \{{SuggestComponent.SuggesterListener.newSearcher()}} calls
\{{suggester.build(core, newSearcher)}} synchronously — see
[SuggestComponent.java|https://github.com/apache/solr/blob/releases/solr/9.10.1/solr/core/src/java/org/apache/solr/handler/component/SuggestComponent.java#L566-L571].
* This listener is invoked as a \{{newSearcherListener}} on \{{SolrCore}}'s
\{{searcherExecutor}}, which is explicitly a *single-threaded* executor (see
the comment "WARNING: this code assumes a single threaded executor" in
\{{SolrCore.java}}). Warmup, all \{{newSearcherListener}}s, and searcher
registration all run sequentially on this one thread.
* \{{DirectUpdateHandler2.commit()}} blocks the committing thread on
\{{waitSearcher[0].get()}} — a no-arg, *no-timeout* \{{Future.get()}} —
whenever \{{waitSearcher=true}}, which is the default for a normal
\{{commit=true}} request (\{{CommitUpdateCommand.waitSearcher}} defaults to
\{{true}}).
* Net effect: an arbitrarily slow \{{buildOnCommit}} rebuild (or any other slow
\{{newSearcherListener}}) blocks the commit-issuing thread/HTTP request
indefinitely, bounded only by client/proxy socket timeouts.
This exact hazard is already flagged in the reference guide ("Some lookup
implementations may take a long time to build... using buildOnCommit or
buildOnOptimize, particularly with a high frequency of soft commits is not
recommended") but the current behavior is worse than "slow" — it can hang a
commit indefinitely with no internal safeguard.
Note: \{{SpellCheckComponent}} has an analogous
\{{buildOnCommit}}/\{{SpellCheckerListener}} pattern and likely shares this
same architectural risk; filing this ticket scoped to \{{SuggestComponent}}
only.
h3. Reproduction
Two JUnit tests demonstrate this deterministically using a test-only dictionary
that simulates slow I/O (sleeping a fixed time per term, so total build time is
bounded and reproducible):
* \{{SuggestComponentBuildOnCommitBlocksCommitTest}} — with
\{{buildOnCommit=true}}, \{{commit()}} takes as long as the (artificially slow)
suggester build.
* \{{SuggestComponentBuildOnCommitDisabledCommitStaysFastTest}} — same slow
dictionary, \{{buildOnCommit=false}}: \{{commit()}} stays fast, isolating the
cause to \{{buildOnCommit}} itself.
h3. Suggested direction (open to discussion)
* Run \{{buildOnCommit}}/\{{buildOnOptimize}} suggester rebuilds off the shared
single-threaded \{{searcherExecutor}} (e.g. a dedicated executor) so they don't
block warmup/registration/commit for other listeners.
* And/or decouple the rebuild from the commit's \{{waitSearcher}} path
entirely, so a slow rebuild degrades to "suggestions are stale a bit longer"
rather than "commit hangs."
* And/or add an explicit, documented timeout/async option for
\{{buildOnCommit}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
