[
https://issues.apache.org/jira/browse/IMPALA-13475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Csaba Ringhofer updated IMPALA-13475:
-------------------------------------
Description:
KrpcDataStreamRecvr::SenderQueue::ProcessDeferredRpc() can fail to process the
deferred RCP if batch_queue is not empty and the batch queue + the currently
processed batch would consume too much memory (see
KrpcDataStreamRecvr::CanEnqueue for details). The deferred RPC is moved back to
the queue in this case.
Meanwhile KrpcDataStreamRecvr::SenderQueue::GetBatch() doesn't consider the mem
requirement of the batches when initiating the deserialization of deferred RCPs
( EnqueueDeserializeTask) and tries to deserialize as much batches in parallel
as possible (FLAGS_datastream_service_num_deserialization_threads,
https://github.com/apache/impala/blob/c83e5d97693fd3035b33622512d1584a5e56ce8b/be/src/runtime/krpc-data-stream-recvr.cc#L281).
This means that several threads may start ProcessDeferredRpc() even if
GetBatch() could have predicted that most will fail due to the memory limit.
While this ProcessDeferredRpc() will fail early in this case and won't do much
work, these extra failed attempts make
lock contention worse in KrpcDataStreamRecvr::SenderQueue. In the worst case
when only 1 batch fits to memory this can lead to
O(FLAGS_datastream_service_num_deserialization_threads * num_batches) wasted
ProcessDeferredRpc attempts.
was:
KrpcDataStreamRecvr::SenderQueue::ProcessDeferredRpc() can fail to process the
deferred RCP if batch_queue is not empty and the batch queue + the currently
processed batch would consume too much memory (see
KrpcDataStreamRecvr::CanEnqueue for details). The deferred RPC is moved back to
the queue in this case.
Meanwhile KrpcDataStreamRecvr::SenderQueue::GetBatch() doesn't consider the mem
requirement of the batches when initiating the deserialization of deferred RCPs
( EnqueueDeserializeTask) and tries to deserialize as much batches in parallel
as possible (FLAGS_datastream_service_num_deserialization_threads,
https://github.com/apache/impala/blob/c83e5d97693fd3035b33622512d1584a5e56ce8b/be/src/runtime/krpc-data-stream-recvr.cc#L281).
This means that several threads may start ProcessDeferredRpc() even if
GetBatch() could have predicted that most will fail due to the memory limit.
While this ProcessDeferredRpc() will fail early in this case and won't do much
work, these extra failed attempts lock contention worse in
KrpcDataStreamRecvr::SenderQueue. In the worst case when only 1 batch fits to
memory this can lead to O(FLAGS_datastream_service_num_deserialization_threads
* num_batches) wasted ProcessDeferredRpc attempts.
> Consider byte size when enqueuing deferred RPCs in KrpcDataStreamRecvr
> ----------------------------------------------------------------------
>
> Key: IMPALA-13475
> URL: https://issues.apache.org/jira/browse/IMPALA-13475
> Project: IMPALA
> Issue Type: Improvement
> Components: Backend
> Reporter: Csaba Ringhofer
> Priority: Critical
>
> KrpcDataStreamRecvr::SenderQueue::ProcessDeferredRpc() can fail to process
> the deferred RCP if batch_queue is not empty and the batch queue + the
> currently processed batch would consume too much memory (see
> KrpcDataStreamRecvr::CanEnqueue for details). The deferred RPC is moved back
> to the queue in this case.
> Meanwhile KrpcDataStreamRecvr::SenderQueue::GetBatch() doesn't consider the
> mem requirement of the batches when initiating the deserialization of
> deferred RCPs ( EnqueueDeserializeTask) and tries to deserialize as much
> batches in parallel as possible
> (FLAGS_datastream_service_num_deserialization_threads,
> https://github.com/apache/impala/blob/c83e5d97693fd3035b33622512d1584a5e56ce8b/be/src/runtime/krpc-data-stream-recvr.cc#L281).
> This means that several threads may start ProcessDeferredRpc() even if
> GetBatch() could have predicted that most will fail due to the memory limit.
> While this ProcessDeferredRpc() will fail early in this case and won't do
> much work, these extra failed attempts make
> lock contention worse in KrpcDataStreamRecvr::SenderQueue. In the worst case
> when only 1 batch fits to memory this can lead to
> O(FLAGS_datastream_service_num_deserialization_threads * num_batches) wasted
> ProcessDeferredRpc attempts.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
