[
https://issues.apache.org/jira/browse/SOLR-18348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SOLR-18348:
----------------------------------
Labels: pull-request-available (was: )
> `buildOnCommit=true` suggester rebuild blocks commit (and the client request)
> indefinitely with no timeout
> ----------------------------------------------------------------------------------------------------------
>
> Key: SOLR-18348
> URL: https://issues.apache.org/jira/browse/SOLR-18348
> Project: Solr
> Issue Type: Bug
> Components: Suggester
> Affects Versions: 9.10.1
> Reporter: Eric Pugh
> Assignee: Eric Pugh
> Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> h3. Problem
> When a suggester is configured with `buildOnCommit=true`, a slow suggester
> rebuild does not just make that commit take longer in the background — it
> synchronously blocks the commit call itself, and therefore the client/update
> request waiting on it, with *no timeout*. Under sustained heavy indexing
> combined with throttled storage (e.g. an Azure disk after burst credits are
> exhausted), this manifests as outright request timeouts requiring manual
> operator intervention, rather than graceful degradation.
> A customer hit this in production: ~4 million product documents being indexed
> against a suggester with an I/O-intensive `AnalyzingInfixSuggester`
> configuration, on a disk that had exhausted its burst-credit allowance.
> Indexing requests began timing out. Thread dumps showed threads stuck in:
> {code}
> "searcherExecutor-250-thread-3-processing-...-core_node4" ... RUNNABLE
> at org.apache.lucene.util.compress.LZ4.decompress(LZ4.java:138)
> at
> org.apache.lucene.codecs.lucene90.LZ4WithPresetDictCompressionMode$LZ4WithPresetDictDecompressor.decompress(...)
> at
> org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsReader...
> at
> org.apache.lucene.search.suggest.DocumentDictionary$DocumentInputIterator.next(DocumentDictionary.java:169)
> at
> org.apache.lucene.search.suggest.analyzing.AnalyzingInfixSuggester.build(AnalyzingInfixSuggester.java:346)
> {code}
> h3. Root cause
> * \{{SuggestComponent.SuggesterListener.newSearcher()}} calls
> \{{suggester.build(core, newSearcher)}} synchronously — see
> [SuggestComponent.java|https://github.com/apache/solr/blob/releases/solr/9.10.1/solr/core/src/java/org/apache/solr/handler/component/SuggestComponent.java#L566-L571].
> * This listener is invoked as a \{{newSearcherListener}} on \{{SolrCore}}'s
> \{{searcherExecutor}}, which is explicitly a *single-threaded* executor (see
> the comment "WARNING: this code assumes a single threaded executor" in
> \{{SolrCore.java}}). Warmup, all \{{newSearcherListener}}s, and searcher
> registration all run sequentially on this one thread.
> * \{{DirectUpdateHandler2.commit()}} blocks the committing thread on
> \{{waitSearcher[0].get()}} — a no-arg, *no-timeout* \{{Future.get()}} —
> whenever \{{waitSearcher=true}}, which is the default for a normal
> \{{commit=true}} request (\{{CommitUpdateCommand.waitSearcher}} defaults to
> \{{true}}).
> * Net effect: an arbitrarily slow \{{buildOnCommit}} rebuild (or any other
> slow \{{newSearcherListener}}) blocks the commit-issuing thread/HTTP request
> indefinitely, bounded only by client/proxy socket timeouts.
> This exact hazard is already flagged in the reference guide ("Some lookup
> implementations may take a long time to build... using buildOnCommit or
> buildOnOptimize, particularly with a high frequency of soft commits is not
> recommended") but the current behavior is worse than "slow" — it can hang a
> commit indefinitely with no internal safeguard.
> Note: \{{SpellCheckComponent}} has an analogous
> \{{buildOnCommit}}/\{{SpellCheckerListener}} pattern and likely shares this
> same architectural risk; filing this ticket scoped to \{{SuggestComponent}}
> only.
> h3. Reproduction
> Two JUnit tests demonstrate this deterministically using a test-only
> dictionary that simulates slow I/O (sleeping a fixed time per term, so total
> build time is bounded and reproducible):
> * \{{SuggestComponentBuildOnCommitBlocksCommitTest}} — with
> \{{buildOnCommit=true}}, \{{commit()}} takes as long as the (artificially
> slow) suggester build.
> * \{{SuggestComponentBuildOnCommitDisabledCommitStaysFastTest}} — same slow
> dictionary, \{{buildOnCommit=false}}: \{{commit()}} stays fast, isolating the
> cause to \{{buildOnCommit}} itself.
> h3. Suggested direction (open to discussion)
> * Run \{{buildOnCommit}}/\{{buildOnOptimize}} suggester rebuilds off the
> shared single-threaded \{{searcherExecutor}} (e.g. a dedicated executor) so
> they don't block warmup/registration/commit for other listeners.
> * And/or decouple the rebuild from the commit's \{{waitSearcher}} path
> entirely, so a slow rebuild degrades to "suggestions are stale a bit longer"
> rather than "commit hangs."
> * And/or add an explicit, documented timeout/async option for
> \{{buildOnCommit}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
