Artem Abeleshev created SOLR-17707:
--------------------------------------
Summary: HttpJdkSolrClient threads stuck on exceptions
Key: SOLR-17707
URL: https://issues.apache.org/jira/browse/SOLR-17707
Project: Solr
Issue Type: Bug
Security Level: Public (Default Security Level. Issues are Public)
Components: SolrJ
Affects Versions: 9.6.1
Environment: There is a problem with _HttpJdkSolrClient_ used by
{_}SolrJ{_}. In case of exception occurs on sending data (in my case it was
{_}connection reset by peer{_}) the thread from the http client pool get stuck.
Internally, _HttpJdkSolrClient_ using _PipedOutputStream_ and
_PipedInputStream_ and in case of exception it ends up waiting forever. Seems
like _PipedOutputStream_ buffer remains full and _PipedInputStream_ stuck in a
waiting state waiting for the avaialble space to write. This will consume one
thread from the pool making it blocked. By default, _HttpJdkSolrClient_ created
with an executor of a _4_ threads, if all of them are blocked the _SolrJ_ will
be unable to work.
There seems to be no problem with that if switching to use _Http2SolrClient_
instead of _HttpJdkSolrClient_ as _Http2SolrClient_ doesn't use
_PipedOutputStream_ and {_}PipedInputStream{_}.
Here is a part of threaddump showing the http client threads get stuck:
{code:java}
"HttpJdkSolrClient-1-thread-1-processing-system
ad571f96-762d-4e9a-8f9c-f5580413397d" prio=5 Id=52 TIMED_WAITING on
java.io.PipedInputStream@67168d76
at java.base@17.0.12/java.lang.Object.wait(Native Method)
- waiting on java.io.PipedInputStream@67168d76
at java.base@17.0.12/java.io.PipedInputStream.awaitSpace(Unknown Source)
at java.base@17.0.12/java.io.PipedInputStream.receive(Unknown Source)
at java.base@17.0.12/java.io.PipedOutputStream.write(Unknown Source)
at
org.apache.solr.common.util.FastOutputStream.flush(FastOutputStream.java:207)
at
org.apache.solr.common.util.FastOutputStream.flushBuffer(FastOutputStream.java:200)
at org.apache.solr.common.util.JavaBinCodec.marshal(JavaBinCodec.java:170)
at
org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec.marshal(JavaBinUpdateRequestCodec.java:100)
...
Number of locked synchronizers = 1
- java.util.concurrent.ThreadPoolExecutor$Worker@bdec8d6
"HttpJdkSolrClient-1-thread-2-processing-system
ad571f96-762d-4e9a-8f9c-f5580413397d" prio=5 Id=53 TIMED_WAITING on
java.io.PipedInputStream@571e7f87
at java.base@17.0.12/java.lang.Object.wait(Native Method)
- waiting on java.io.PipedInputStream@571e7f87
at java.base@17.0.12/java.io.PipedInputStream.awaitSpace(Unknown Source)
at java.base@17.0.12/java.io.PipedInputStream.receive(Unknown Source)
at java.base@17.0.12/java.io.PipedOutputStream.write(Unknown Source)
at
org.apache.solr.common.util.FastOutputStream.flush(FastOutputStream.java:207)
at
org.apache.solr.common.util.FastOutputStream.flushBuffer(FastOutputStream.java:200)
at org.apache.solr.common.util.JavaBinCodec.marshal(JavaBinCodec.java:170)
at
org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec.marshal(JavaBinUpdateRequestCodec.java:100)
...
Number of locked synchronizers = 1
- java.util.concurrent.ThreadPoolExecutor$Worker@7e6cf6f2
"HttpJdkSolrClient-1-thread-3-processing-system
ad571f96-762d-4e9a-8f9c-f5580413397d" prio=5 Id=54 TIMED_WAITING on
java.io.PipedInputStream@497c4c19
at java.base@17.0.12/java.lang.Object.wait(Native Method)
- waiting on java.io.PipedInputStream@497c4c19
at java.base@17.0.12/java.io.PipedInputStream.awaitSpace(Unknown Source)
at java.base@17.0.12/java.io.PipedInputStream.receive(Unknown Source)
at java.base@17.0.12/java.io.PipedOutputStream.write(Unknown Source)
at
org.apache.solr.common.util.FastOutputStream.flush(FastOutputStream.java:207)
at
org.apache.solr.common.util.FastOutputStream.flushBuffer(FastOutputStream.java:200)
at org.apache.solr.common.util.JavaBinCodec.marshal(JavaBinCodec.java:170)
at
org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec.marshal(JavaBinUpdateRequestCodec.java:100)
...
Number of locked synchronizers = 1
- java.util.concurrent.ThreadPoolExecutor$Worker@7a2b5f30
"HttpJdkSolrClient-1-thread-4-processing-system
ad571f96-762d-4e9a-8f9c-f5580413397d" prio=5 Id=55 TIMED_WAITING on
java.io.PipedInputStream@77907562
at java.base@17.0.12/java.lang.Object.wait(Native Method)
- waiting on java.io.PipedInputStream@77907562
at java.base@17.0.12/java.io.PipedInputStream.awaitSpace(Unknown Source)
at java.base@17.0.12/java.io.PipedInputStream.receive(Unknown Source)
at java.base@17.0.12/java.io.PipedOutputStream.write(Unknown Source)
at
org.apache.solr.common.util.FastOutputStream.flush(FastOutputStream.java:207)
at
org.apache.solr.common.util.FastOutputStream.write(FastOutputStream.java:97)
at org.apache.solr.common.util.JavaBinCodec.writeStr(JavaBinCodec.java:916)
at
org.apache.solr.common.util.JavaBinCodec.writePrimitive(JavaBinCodec.java:1062)
...
Number of locked synchronizers = 1
- java.util.concurrent.ThreadPoolExecutor$Worker@2fb4c889 {code}
Reporter: Artem Abeleshev
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org
