[
https://issues.apache.org/jira/browse/IMPALA-13147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18045300#comment-18045300
]
ASF subversion and git services commented on IMPALA-13147:
----------------------------------------------------------
Commit c3dc7f9667564a4ffb9644e6f3d0103c007cdc10 in impala's branch
refs/heads/master from Michael Smith
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=c3dc7f966 ]
IMPALA-13147: Limit concurrency of link jobs
Configure separate compile and link pools for ninja. Configures link
parallelism based on expected memory use, which can be reduced by
setting IMPALA_MINIMAL_DEBUG_INFO=true or IMPALA_SPLIT_DEBUG_INFO=true.
Adds IMPALA_MAKE_CMD to simplify using the ninja build tool for all make
operations in scripts. Install ninja on Ubuntu. Adds a '-make' option to
buildall.sh to force using 'make'.
Adds MOLD_JOBS=1 to avoid overloading the system when trying 'mold' and
linking test binaries. However 'mold' is not selected as the default
due to test failures around SASL/GSSAPI (see IMPALA-14527).
Switches bin/jenkins/all-tests.sh to use ninja and removes the guard in
bootstrap_development.sh limiting IMPALA_BUILD_THREADS as it's no longer
needed with ninja.
SKIP_BE_TEST_PATTERN in run-backend-tests is unused (only used with
TARGET_FILESYSTEM=local) so I don't attempt to make it work with ninja.
Tested with local 'IMPALA_SPLIT_DEBUG_INFO=true buildall.sh -skiptests'
with default (make) and IMPALA_MAKE_CMD=ninja.
Change-Id: I0952dc19ace5c9c42bed0d2ffb61499656c0a2db
Reviewed-on: http://gerrit.cloudera.org:8080/23572
Reviewed-by: Joe McDonnell <[email protected]>
Reviewed-by: Pranav Lodha <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> Add support for limiting the concurrency of link jobs
> -----------------------------------------------------
>
> Key: IMPALA-13147
> URL: https://issues.apache.org/jira/browse/IMPALA-13147
> Project: IMPALA
> Issue Type: Improvement
> Components: Infrastructure
> Affects Versions: Impala 4.5.0
> Reporter: Joe McDonnell
> Assignee: Michael Smith
> Priority: Major
> Fix For: Impala 5.0.0
>
>
> Link jobs can use a lot of memory due to the amount of debug info. The level
> of concurrency that is useful for compilation can be too high for linking.
> Running a link-heavy command like buildall.sh -skiptests can run out of
> memory from linking all of the backend tests / benchmarks.
> It would be useful to be able to limit the number of concurrent link jobs.
> There are two basic approaches:
> When using the ninja generator for CMake, ninja supports having job pools
> with limited parallelism. CMake has support for mapping link tasks to their
> own pool. Here is an example:
> {noformat}
> set(CMAKE_JOB_POOLS compilation_pool=24 link_pool=8)
> set(CMAKE_JOB_POOL_COMPILE compilation_pool)
> set(CMAKE_JOB_POOL_LINK link_pool){noformat}
> The makefile generator does not have equivalent functionality, but we could
> do a more limited version where buildall.sh can split the -skiptests into two
> make invocations. The first does all the compilation with full parallelism
> (equivalent to -notests) and then the second make invocation does the backend
> tests / benchmarks with a reduced parallelism.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
