[
https://issues.apache.org/jira/browse/HTTPCORE-773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17902945#comment-17902945
]
Oleg Kalnichevski commented on HTTPCORE-773:
--------------------------------------------
> Regarding runtime stats, What do you think about exposing pending requests on
> a individual IO worker threads? This will help get visibility on when the
> system is behaving in a degraded way where one or more of the threads are
> stuck and if more requests keep getting assigned to them, there is no
> possibility of recovering such threads
[~anithai1209] Please note that i/o dispatch threads manage _multiple_ channels
simultaneously. Each channel is effectively a separate connection with its own
request queue. Collecting stats of pending request on a per i/o dispatch basis
would effectively mean having to iterate over all channels managed by the
dispatch worker, extracting the specific protocol handler for the channel and
counting protocol specific requests still pending.
1. This would be very expensive
2. It would also violate the principle of separation of concerns and make the
i/o reactor layer tightly coupled to a single protocol handler. While we are an
HTTP focused project it is still nasty from the design standpoint.
3. last select time (already exposed in the stats) is by far better metric to
detect i/o inactivity and it also virtually has no performance overhead.
Oleg
> Safeguard against blocked IOWorker thread
> -----------------------------------------
>
> Key: HTTPCORE-773
> URL: https://issues.apache.org/jira/browse/HTTPCORE-773
> Project: HttpComponents HttpCore
> Issue Type: Improvement
> Components: HttpCore NIO
> Reporter: Anithamahalakshmi
> Priority: Major
> Fix For: 5.4-alpha1
>
>
> # During the initialisation of ApacheAsyncClient, a single main thread is
> created -
> [https://github.com/xSke/CoreServer/blob/master/src/org/apache/http/impl/nio/client/CloseableHttpAsyncClientBase.java#L40.]
> # This IO reactor thread does two key jobs: 1) It internally instantiate IO
> dispatcher threads once They never stop either. They keep polling for new IO
> event) and 2) It then works as a daemon (which never stops) to process new
> events. It worth to notice that all events IO reactor thread is connection
> related event per
> https://github.com/wso2/wso2-httpcore-nio/blob/master/modules/httpcore-nio/src/main/java/org/apache/http/impl/nio/reactor/DefaultConnectingIOReactor.java#L164
> # For http request which needs a new IO connection, IO reactor thread is
> invoked to create such IO connection asynchronously -
> [https://github.com/wso2/wso2-httpcore-nio/blob/master/modules/httpcore-nio/src/main/java/org/apache/http/nio/pool/AbstractNIOConnPool.java#L427]
> Connection pending to be created in such asynchronous process is reported as
> {{pending}} connection in Apache connection status.
> # In such asynchronous process, connection to be created is added into
> {{requestQueue}} -
> [https://github.com/wso2/wso2-httpcore-nio/blob/master/modules/httpcore-nio/src/main/java/org/apache/http/impl/nio/reactor/DefaultConnectingIOReactor.java#L251].
> {{requestQueue}} is later on polled by IO reactor thread as a daemon, and
> established channel should be handed off to IO dispatcher thread via
> following logic
> -[https://github.com/wso2/wso2-httpcore-nio/blob/master/modules/httpcore-nio/src/main/java/org/apache/http/impl/nio/reactor/DefaultConnectingIOReactor.java#L276].
> It is interesting to notice that which IO dispatcher thread such new channel
> should be handed off to is based on round-robin per
> [https://github.com/wso2/wso2-httpcore-nio/blob/master/modules/httpcore-nio/src/main/java/org/apache/http/impl/nio/reactor/AbstractMultiworkerIOReactor.java#L475.]
>
> # The IO dispatcher thread is expected to pull and process new IO event
> assigned to it by selector, and then process assigned new channel as well per
> [https://github.com/wso2/wso2-httpcore-nio/blob/master/modules/httpcore-nio/src/main/java/org/apache/http/impl/nio/reactor/AbstractIOReactor.java#L222]
> However if a certain IO dispatcher thread is blocked while some previous
> processing, it goes undetected and new requests+connection pair keep getting
> assigned to the blocked thread which is unable to process the request and
> release the connection back to the pool, eventually we exhaust the max
> connection possible from the pool and cause availability drop.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@hc.apache.org
For additional commands, e-mail: dev-h...@hc.apache.org
