[
https://issues.apache.org/jira/browse/IMPALA-10204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18030449#comment-18030449
]
Yida Wu commented on IMPALA-10204:
----------------------------------
The admissiond process can show a significant increase in memory usage when a
high-concurrency workload running, which contains complex queries. This can be
reproduced by running below script that executes the attached query_tpcds.sql
when the AC service is enabled.
{code:java}
for i in {1..100}; do
bin/impala-shell.sh -i [ip_addr] -f query_tpcds.sql &
sleep 1
done {code}
> Evaluate AdmitQuery params for efficiency
> -----------------------------------------
>
> Key: IMPALA-10204
> URL: https://issues.apache.org/jira/browse/IMPALA-10204
> Project: IMPALA
> Issue Type: Sub-task
> Components: Distributed Exec
> Reporter: Thomas Tauber-Marshall
> Priority: Critical
> Attachments: query_tpcds.sql
>
>
> In the first version of the AdmissionControlService, we're sending the entire
> TQueryExecRequest/TQueryOptions as a sidecar to the admission controller.
> There are various things contained in the TQueryExecRequest/TQueryOptions
> that are not actually needed by the admission controller, and sending them
> increases network load and query running time unnecessarily.
> We should evaluate how much of a perf impact there is due to this and how
> much could actually be removed.
> Some small things may be non-trivial to remove and ultimately not worth it,
> for example the tree of TPlanNodes contains some info needed by the admission
> controller (eg. memory estimates) and some things that are not (eg. runtime
> filter descriptors). Making two parallel trees, one with only
> admission-required data (which would require extensive refactoring in the
> planner or wasted work in the coordinator copying out the required parts from
> what the planner returns) may be too complicated/introduce too much other
> overhead to be worth it.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
