[
https://issues.apache.org/jira/browse/IMPALA-13147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18031985#comment-18031985
]
Quanlong Huang edited comment on IMPALA-13147 at 10/22/25 7:58 AM:
-------------------------------------------------------------------
I think we need to prioritize this. The all-build-options-ub2004 Jenkins job
consistently fail on my patch:
[https://jenkins.impala.io/job/all-build-options-ub2004/3311]
{noformat}
collect2: fatal error: ld terminated with signal 9 [Killed]
compilation terminated.
Generated:
/home/ubuntu/Impala/logs/extra_junit_xml_logs/generate_junitxml.build.0afa75ad1e8c4486b83f0be965c7337a.20251021_02_56_48.xml
make[2]: *** [be/src/rpc/CMakeFiles/rpc-mgr-kerberized-test.dir/build.make:265:
be/build/debug/rpc/rpc-mgr-kerberized-test] Error 1
make[2]: *** Deleting file 'be/build/debug/rpc/rpc-mgr-kerberized-test'
make[1]: *** [CMakeFiles/Makefile2:9303:
be/src/rpc/CMakeFiles/rpc-mgr-kerberized-test.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....{noformat}
[https://jenkins.impala.io/job/all-build-options-ub2004/3314]
{noformat}
collect2: fatal error: ld terminated with signal 9 [Killed]
compilation terminated.
Generated:
/home/ubuntu/Impala/logs/extra_junit_xml_logs/generate_junitxml.build.d5ff6086cd9b18d60420f18e2a1c0020.20251022_01_11_23.xml
make[2]: ***
[be/src/runtime/bufferpool/CMakeFiles/buffer-pool-test.dir/build.make:261:
be/build/debug/runtime/bufferpool/buffer-pool-test] Error 1
make[2]: *** Deleting file 'be/build/debug/runtime/bufferpool/buffer-pool-test'
make[1]: *** [CMakeFiles/Makefile2:7178:
be/src/runtime/bufferpool/CMakeFiles/buffer-pool-test.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....{noformat}
The build on master branch also failed recently:
[https://jenkins.impala.io/job/all-build-options-ub2004/3288]
{noformat}
../bin/mvn-quiet.sh: line 49: 37614 Killed mvn -B
$IMPALA_MAVEN_OPTIONS $LOGGING_OPTIONS "$@"
mvn -U -U -B dependency:tree install -DskipTests exited with code 0{noformat}
The failed build option in these jobs are all
{noformat}
-skiptests -noclean USE_APACHE_COMPONENTS=false {noformat}
CC [~MikaelSmith]
was (Author: stiga-huang):
I think we need to prioritize this. The all-build-options-ub2004 Jenkins job
consistently fail on my patch:
[https://jenkins.impala.io/job/all-build-options-ub2004/3311]
{noformat}
collect2: fatal error: ld terminated with signal 9 [Killed]
compilation terminated.
Generated:
/home/ubuntu/Impala/logs/extra_junit_xml_logs/generate_junitxml.build.0afa75ad1e8c4486b83f0be965c7337a.20251021_02_56_48.xml
make[2]: *** [be/src/rpc/CMakeFiles/rpc-mgr-kerberized-test.dir/build.make:265:
be/build/debug/rpc/rpc-mgr-kerberized-test] Error 1
make[2]: *** Deleting file 'be/build/debug/rpc/rpc-mgr-kerberized-test'
make[1]: *** [CMakeFiles/Makefile2:9303:
be/src/rpc/CMakeFiles/rpc-mgr-kerberized-test.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....{noformat}
[https://jenkins.impala.io/job/all-build-options-ub2004/3314]
{noformat}
collect2: fatal error: ld terminated with signal 9 [Killed]
compilation terminated.
Generated:
/home/ubuntu/Impala/logs/extra_junit_xml_logs/generate_junitxml.build.d5ff6086cd9b18d60420f18e2a1c0020.20251022_01_11_23.xml
make[2]: ***
[be/src/runtime/bufferpool/CMakeFiles/buffer-pool-test.dir/build.make:261:
be/build/debug/runtime/bufferpool/buffer-pool-test] Error 1
make[2]: *** Deleting file 'be/build/debug/runtime/bufferpool/buffer-pool-test'
make[1]: *** [CMakeFiles/Makefile2:7178:
be/src/runtime/bufferpool/CMakeFiles/buffer-pool-test.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....{noformat}
The build on master branch also failed recently:
[https://jenkins.impala.io/job/all-build-options-ub2004/3288]
{noformat}
../bin/mvn-quiet.sh: line 49: 37614 Killed mvn -B
$IMPALA_MAVEN_OPTIONS $LOGGING_OPTIONS "$@"
mvn -U -U -B dependency:tree install -DskipTests exited with code 0{noformat}
CC [~MikaelSmith]
> Add support for limiting the concurrency of link jobs
> -----------------------------------------------------
>
> Key: IMPALA-13147
> URL: https://issues.apache.org/jira/browse/IMPALA-13147
> Project: IMPALA
> Issue Type: Improvement
> Components: Infrastructure
> Affects Versions: Impala 4.5.0
> Reporter: Joe McDonnell
> Priority: Major
>
> Link jobs can use a lot of memory due to the amount of debug info. The level
> of concurrency that is useful for compilation can be too high for linking.
> Running a link-heavy command like buildall.sh -skiptests can run out of
> memory from linking all of the backend tests / benchmarks.
> It would be useful to be able to limit the number of concurrent link jobs.
> There are two basic approaches:
> When using the ninja generator for CMake, ninja supports having job pools
> with limited parallelism. CMake has support for mapping link tasks to their
> own pool. Here is an example:
> {noformat}
> set(CMAKE_JOB_POOLS compilation_pool=24 link_pool=8)
> set(CMAKE_JOB_POOL_COMPILE compilation_pool)
> set(CMAKE_JOB_POOL_LINK link_pool){noformat}
> The makefile generator does not have equivalent functionality, but we could
> do a more limited version where buildall.sh can split the -skiptests into two
> make invocations. The first does all the compilation with full parallelism
> (equivalent to -notests) and then the second make invocation does the backend
> tests / benchmarks with a reduced parallelism.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
