[
https://issues.apache.org/jira/browse/IMPALA-14493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18032506#comment-18032506
]
ASF subversion and git services commented on IMPALA-14493:
----------------------------------------------------------
Commit 1bc7cdbff6bfe468254ba0c507e0a287ae46e6c8 in impala's branch
refs/heads/master from Yida Wu
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=1bc7cdbff ]
IMPALA-14493: Cap memory usage of global admission service
The global admission service can experience OOM errors under
high concurrency because its process memory tracker is inaccurate
and doesn't account for all memory allocations.
Ensuring memory tracker accurately accounts for every allocation
could be difficult, this patch uses a simpler solution to
introduce a hard memory cap using tcmalloc statistics, which
accurately reflect the true process memory usage. If a new query
is submitted while tcmalloc memory usage is over the process
limit, the query will be rejected immediately to protect from OOM.
Adds a new flag enable_admission_service_mem_safeguard allowing
this feature to be enabled or disabled. By default, this feature is
turned on
Tests:
Added test test_admission_service_low_mem_limit.
Passed exhaustive tests.
Change-Id: I2ee2c942a73fcd69358851fc2fdc0fc4fe531c73
Reviewed-on: http://gerrit.cloudera.org:8080/23542
Reviewed-by: Abhishek Rawat <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> Capping Memory Usage of Global Admission Service
> -------------------------------------------------
>
> Key: IMPALA-14493
> URL: https://issues.apache.org/jira/browse/IMPALA-14493
> Project: IMPALA
> Issue Type: Bug
> Reporter: Yida Wu
> Assignee: Yida Wu
> Priority: Major
>
> The admission service ({{{}admissiond{}}}) can OOM on high workload because
> its process memory tracker is inaccurate and doesn't account for all memory
> allocations.
> Ensuring memory tracker accurately accounts for every allocation could be
> difficult, one simpler solution is to introduce a hard memory cap using
> {{tcmalloc}} statistics, which accurately reflect the true process memory
> usage. If a new query is submitted while {{tcmalloc}} memory usage is over
> the process limit, the query will be rejected immediately to protect from OOM.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
