[
https://issues.apache.org/jira/browse/IMPALA-14445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18021904#comment-18021904
]
Joe McDonnell commented on IMPALA-14445:
----------------------------------------
The executor can be rejecting the message due to hitting a memory limit, so we
should make clear which memory limit it is hitting. This may be completely
unrelated to the sizes of the control service queue / data stream service
queue, as it may be hitting the process memory limit.
> Improve error message for RPC backpressure and other KRPC error messages
> ------------------------------------------------------------------------
>
> Key: IMPALA-14445
> URL: https://issues.apache.org/jira/browse/IMPALA-14445
> Project: IMPALA
> Issue Type: Task
> Components: Distributed Exec
> Affects Versions: Impala 5.0.0
> Reporter: Joe McDonnell
> Priority: Critical
>
> On a cluster, we saw queries failing with errors like this:
> {noformat}
> Exec() rpc failed: Remote error: Service unavailable: ExecQueryFInstances
> request on impala.ControlService from 1.2.3.4:12345 dropped due to
> backpressure. The service queue contains 0 items out of a maximum of
> 2147483647; memory consumption is 0.{noformat}
> In this case, the executor is rejecting a message from the coordinator
> (1.2.3.4:12345 is the IP:port of a coordinator) and the query is failing due
> to that. The error message doesn't make it clear that the executor is the one
> rejecting the message, so this can lead to focusing on the coordinator rather
> than tracking down the executor. Also, the message should include the
> executor hostname to make it easy to know where to look.
> The executor black-listing / query retry logic constructs a better message
> for the retryable status, but we don't use that for backpressure.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
