kevin-wu24 commented on PR #22111: URL: https://github.com/apache/kafka/pull/22111#issuecomment-4723892565
> Partial review, getting up to speed on the issue again. > > I do wonder if there's a separate problem regarding timing that is worth fixing. > > We should have the following two distinct paths: > > * request times out (no response) → RequestManager moves state to `READY` → retry immediately > * request errors → RequestManager moves state to `BACKING_OFF` for `retryBackoffMs` > > When a fetch can't be sent (e.g. unreachable endpoint), the network channel returns a `BROKER_NOT_AVAILABLE` response, which is arguably meant to push RequestManager into `BACKING_OFF` state. But that response is always discarded, and I think that's the separate issue we might want to fix (in a separate PR). > > RequestManager's response timeout fires at `>= lastSendTimeMs + requestTimeoutMs` while the network channel's synthesized `BROKER_NOT_AVAILABLE` response will only appear at `> createdTimeMs + requestTimeoutMs`. `lastSendTimeMs` equals `createdTimeMs` and both RequestManager and the network channel use the same `requestTimeoutMs` so the RequestManager timeout will always fire first and we'll never actually see the `BROKER_NOT_AVAILABLE` response. Thanks for the review @ahuang98. WRT to your comment: The case with which this case is dealing is one where a TCP handshake between the local and destination node cannot be established. This is evidenced by repeated logs like the one below from the cluster where this behavior was observed: ``` [WARN] 2026-04-06 21:18:28,964 [kafka-0-raft-outbound-request-thread] org.apache.kafka.clients.NetworkClient initiateConnect - [RaftManager id=0] Error connecting to node ... ``` This means the KRaft fetch request is never sent over the wire by the local node. Instead, the connection failure will result in the connection state being `DISCONNECTED`, and `InterBrokerSendThread#checkDisconnects` will complete the request. The specific completion handler being invoked, `KafkaNetworkChannel#sendOnComplete`, is what adds the `BROKER_NOT_AVAILABLE` error response to the local node's message queue. I think you're referring to a case where we fall through to `InterBrokerSendThread#failExpiredRequests`, but I believe our request has already been removed from `unsentRequests` because the connection state to the destination is disconnected. Is my understanding correct? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
