Robert Lucarini created SOLR-18488:
--------------------------------------
Summary: HttpJettySolrClient: selector work is scheduled on the
shared client executor instead of a dedicated thread, adding latency under load
Key: SOLR-18488
URL: https://issues.apache.org/jira/browse/SOLR-18488
Project: Solr
Issue Type: Improvement
Components: clients - java
Affects Versions: 10.0
Reporter: Robert Lucarini
HttpJettySolrClient's ManagedSelector does not run its NIO select loop on a
dedicated thread. Instead, each selector's select() call is "produced" as a
task via AdaptiveExecutionStrategy and dispatched onto the same general-purpose
executor pool used for all other client work (connection setup, request
writing, response parsing). Confirmed via live thread dumps under sustained
concurrent load: with the default useHttp1_1(true) configuration (2 selectors),
both selector threads consistently appear as h2sc-* executor-pool threads
running ManagedSelector$SelectorProducer.produce() →
AdaptiveExecutionStrategy.produceTask(), not as separately-named,
always-running threads.
This differs from HttpJdkSolrClient's underlying
jdk.internal.net.http.HttpClientImpl, which creates exactly one dedicated
SelectorManager thread at construction time that runs continuously for the life
of the client and never competes with request-processing work for a thread-pool
slot.
Under high-concurrency load (60 VUs, small/frequent requests), this appears to
add a measurable, consistent latency overhead compared to HttpJdkSolrClient —
real A/B benchmark, same workload, same cluster, 5 reps: p99 latency increased
61-119% per rep (overall +68.8%, CI [35.6%, 99.2%]) when switching from
HttpJdkSolrClient to HttpJettySolrClient, with no other configuration changes.
This is plausibly explained by the extra "produce → wait for a free executor
thread to pick up the select task" dispatch hop occurring on every
I/O-readiness cycle, which the JDK client's dedicated-thread design avoids
entirely.
There is currently no way to configure this: the ClientConnector/selector
construction happens inside HttpJettySolrClient.createHttpClient(), which is
private, and HttpJettySolrClient.Builder exposes no method to control selector
count, execution strategy, or whether selection runs on a dedicated thread vs.
the shared executor.
Ask: expose a Builder option to either (a) increase the selector count
independently of useHttp1_1, or (b) run selection on a dedicated thread rather
than the shared client executor — similar in spirit to the existing
useHttp1_1/withMaxConnectionsPerHost builder options.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
