[
https://issues.apache.org/jira/browse/IMPALA-13479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17948778#comment-17948778
]
ASF subversion and git services commented on IMPALA-13479:
----------------------------------------------------------
Commit 1157d6e10f59a87bdb76f8e07f163743b6ef7094 in impala's branch
refs/heads/master from Joe McDonnell
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=1157d6e10 ]
IMPALA-13479: Patch gperftools to remove 1GB limit on thread caches
Upstream gperftools does not allow setting tcmalloc.max_total_thread_cache_bytes
to greater than 1GB. This moves to a new toolchain that has patched
gperftools to remove this limitation and allow setting
tcmalloc.max_total_thread_cache_bytes > 1GB. This also reads back the
value from tcmalloc and prints a warning if it doesn't match what we set.
Testing:
- Set tcmalloc_max_total_thread_cache_bytes to 2GB and verified that
the warning message doesn't appear. On unpatched versions of
gperftools, the warning message does appear.
Change-Id: If78c8734c704090c12737a8c2a8456b73ea4b8e8
Reviewed-on: http://gerrit.cloudera.org:8080/22834
Reviewed-by: Michael Smith <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
Reviewed-by: Joe McDonnell <[email protected]>
> Patch gperftools to allow max_total_thread_cache_bytes to exceed 1GB
> --------------------------------------------------------------------
>
> Key: IMPALA-13479
> URL: https://issues.apache.org/jira/browse/IMPALA-13479
> Project: IMPALA
> Issue Type: Improvement
> Components: Infrastructure
> Affects Versions: Impala 4.5.0
> Reporter: Joe McDonnell
> Priority: Critical
>
> gperftools limits max_total_thread_cache_bytes to 1GB here:
> [https://github.com/gperftools/gperftools/blob/gperftools-2.10/src/thread_cache.cc#L520-L523]
> {noformat}
> void ThreadCache::set_overall_thread_cache_size(size_t new_size) {
> // Clip the value to a reasonable range
> if (new_size < kMinThreadCacheSize) new_size = kMinThreadCacheSize;
> if (new_size > (1<<30)) new_size = (1<<30); // Limit to 1GB{noformat}
> I confirmed that setting --tcmalloc_max_total_thread_cache_bytes=2147483648
> still results in a 1GB limit.
> Sometimes, we would want a higher limit for systems with a large amount of
> memory and CPUs. For example, some systems now have 1TB of memory and 96
> CPUs. With high concurrency, there is high contention on tcmalloc locks on
> central data structures. Increasing the total thread cache size could avoid
> this, and a value higher than 1GB is still a small part of system memory.
> We can patch our toolchain gperftools to allow a higher value (and notify
> gperftools community).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
