[
https://issues.apache.org/jira/browse/IMPALA-14527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18045301#comment-18045301
]
ASF subversion and git services commented on IMPALA-14527:
----------------------------------------------------------
Commit c3dc7f9667564a4ffb9644e6f3d0103c007cdc10 in impala's branch
refs/heads/master from Michael Smith
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=c3dc7f966 ]
IMPALA-13147: Limit concurrency of link jobs
Configure separate compile and link pools for ninja. Configures link
parallelism based on expected memory use, which can be reduced by
setting IMPALA_MINIMAL_DEBUG_INFO=true or IMPALA_SPLIT_DEBUG_INFO=true.
Adds IMPALA_MAKE_CMD to simplify using the ninja build tool for all make
operations in scripts. Install ninja on Ubuntu. Adds a '-make' option to
buildall.sh to force using 'make'.
Adds MOLD_JOBS=1 to avoid overloading the system when trying 'mold' and
linking test binaries. However 'mold' is not selected as the default
due to test failures around SASL/GSSAPI (see IMPALA-14527).
Switches bin/jenkins/all-tests.sh to use ninja and removes the guard in
bootstrap_development.sh limiting IMPALA_BUILD_THREADS as it's no longer
needed with ninja.
SKIP_BE_TEST_PATTERN in run-backend-tests is unused (only used with
TARGET_FILESYSTEM=local) so I don't attempt to make it work with ninja.
Tested with local 'IMPALA_SPLIT_DEBUG_INFO=true buildall.sh -skiptests'
with default (make) and IMPALA_MAKE_CMD=ninja.
Change-Id: I0952dc19ace5c9c42bed0d2ffb61499656c0a2db
Reviewed-on: http://gerrit.cloudera.org:8080/23572
Reviewed-by: Joe McDonnell <[email protected]>
Reviewed-by: Pranav Lodha <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> Switch to mold as the default linker
> ------------------------------------
>
> Key: IMPALA-14527
> URL: https://issues.apache.org/jira/browse/IMPALA-14527
> Project: IMPALA
> Issue Type: Task
> Components: Infrastructure
> Reporter: Michael Smith
> Assignee: Michael Smith
> Priority: Major
>
> The mold linker better utilizes concurrency to speed up linking a single
> binary. This has two benefits
> 1. Development when relinking e.g. impalad will be significantly faster (
> 2. During a full build that links many test binaries, we can use
> MOLD_LINKER=1 to link a single binary at a time - using significantly less
> memory - while still fully utilizing all CPU cores.
> Full builds are also a bit faster. After priming ccache with a full build;
> with mold rebuilds take ~2m33s and memory use remains under 10GiB (24 cores)
> {code}
> $ env MOLD_JOBS=1 IMPALA_LINKER=mold time ./buildall.sh -skiptests
> ...
> 313.23user 67.68system 2:33.65elapsed 247%CPU (0avgtext+0avgdata
> 5664100maxresident)k
> 16inputs+34793584outputs (216major+12078088minor)pagefaults 0swaps
> {code}
> With gold, it takes ~3m15s and maxes out my memory (64GiB) requiring
> additional swap
> {code}
> $ env IMPALA_LINKER=gold time ./buildall.sh -skiptests
> ...
> 261.36user 71.43system 3:15.41elapsed 170%CPU (0avgtext+0avgdata
> 3655132maxresident)k
> 7485904inputs+34794136outputs (88605major+11836509minor)pagefaults 0swaps
> {code}
> Performance of the resulting binaries looks essentially the same:
> https://jenkins.impala.io/job/perf-AB-test-ub2004/336/artifact/Impala/perf_results/latest/performance_result.txt.
> Let's switch to using IMPALA_LINKER=mold by default.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
