[
https://issues.apache.org/jira/browse/CAMEL-24318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aurélien Pupier updated CAMEL-24318:
------------------------------------
Description:
Currently, the build on CI is launched using a single Core. Using multiple
cores should allow to drastically reduce the time.
Locally tests (22 cores, 32Gb RAM, similar to Jenkins large machines):
* -T1C for 642 modules, it took 40'
** command used: ./mvnw clean install -T1C -Dci.env.name=fakeCI-local -B -e
-fae -V -Dnoassembly -Dsurefire.rerunFailingTestsCount=2
-Dfailsafe.rerunFailingTestsCount=1 -Dchetype.test.skip
-Dmaven.test.failure.ignore=true -Dcheckstyle.skip=true install
-Dcamel.threads.virtual.enabled=false -Pcoverage
* without, it took 2h48minutes for the same 642 modules
** command used: ./mvnw clean install -Dci.env.name=fakeCI-local -B -e -fae -V
-Dnoassembly -Dsurefire.rerunFailingTestsCount=2
-Dfailsafe.rerunFailingTestsCount=1 -Dchetype.test.skip
-Dmaven.test.failure.ignore=true -Dcheckstyle.skip=true install
-Dcamel.threads.virtual.enabled=false -Pcoverage
Currently, the full compilation can fail when launched with -T1C
* [ERROR] Failed to execute goal
org.apache.maven.plugins:maven-remote-resources-plugin:3.3.0:process (default)
on project bundles-pom: Error loading supplemental data models: Could not find
resource 'target/legal/legal.xml'. -> [Help 1]
* this one also with single thread locally... [ERROR] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:3.15.0:compile (default-compile)
on project camel-jooq: Compilation failure: Compilation failure:
[ERROR]
/home/apupier/git/camel/components/camel-jooq/target/generated-test-sources/jooq/org/apache/camel/component/jooq/db/tables/daos/AuthorDao.java:[15,52]
package org.springframework.beans.factory.annotation does not exist
[ERROR]
/home/apupier/git/camel/components/camel-jooq/target/generated-test-sources/jooq/org/apache/camel/component/jooq/db/tables/daos/AuthorDao.java:[16,38]
package org.springframework.stereotype does not exist
[ERROR]
/home/apupier/git/camel/components/camel-jooq/target/generated-test-sources/jooq/org/apache/camel/component/jooq/db/tables/daos/AuthorDao.java:[23,2]
cannot find symbol
[ERROR] symbol: class Repository
[ERROR]
/home/apupier/git/camel/components/camel-jooq/target/generated-test-sources/jooq/org/apache/camel/component/jooq/db/tables/daos/BookDao.java:[15,52]
package org.springframework.beans.factory.annotation does not exist
[ERROR]
/home/apupier/git/camel/components/camel-jooq/target/generated-test-sources/jooq/org/apache/camel/component/jooq/db/tables/daos/BookDao.java:[16,38]
package org.springframework.stereotype does not exist
[ERROR]
/home/apupier/git/camel/components/camel-jooq/target/generated-test-sources/jooq/org/apache/camel/component/jooq/db/tables/daos/BookDao.java:[23,2]
cannot find symbol
[ERROR] symbol: class Repository
[ERROR]
/home/apupier/git/camel/components/camel-jooq/target/generated-test-sources/jooq/org/apache/camel/component/jooq/db/tables/daos/BookStoreDao.java:[14,52]
package org.springframework.beans.factory.annotation does not exist
[ERROR]
/home/apupier/git/camel/components/camel-jooq/target/generated-test-sources/jooq/org/apache/camel/component/jooq/db/tables/daos/BookStoreDao.java:[15,38]
package org.springframework.stereotype does not exist
[ERROR]
/home/apupier/git/camel/components/camel-jooq/target/generated-test-sources/jooq/org/apache/camel/component/jooq/db/tables/daos/BookStoreDao.java:[22,2]
cannot find symbol
[ERROR] symbol: class Repository
[ERROR]
/home/apupier/git/camel/components/camel-jooq/target/generated-test-sources/jooq/org/apache/camel/component/jooq/db/tables/daos/BookToBookStoreDao.java:[14,52]
package org.springframework.beans.factory.annotation does not exist
[ERROR]
/home/apupier/git/camel/components/camel-jooq/target/generated-test-sources/jooq/org/apache/camel/component/jooq/db/tables/daos/BookToBookStoreDao.java:[15,38]
package org.springframework.stereotype does not exist
[ERROR]
/home/apupier/git/camel/components/camel-jooq/target/generated-test-sources/jooq/org/apache/camel/component/jooq/db/tables/daos/BookToBookStoreDao.java:[22,2]
cannot find symbol
[ERROR] symbol: class Repository
[ERROR]
/home/apupier/git/camel/components/camel-jooq/target/generated-test-sources/jooq/org/apache/camel/component/jooq/db/tables/daos/AuthorDao.java:[36,6]
cannot find symbol
[ERROR] symbol: class Autowired
[ERROR] location: class org.apache.camel.component.jooq.db.tables.daos.AuthorDao
[ERROR]
/home/apupier/git/camel/components/camel-jooq/target/generated-test-sources/jooq/org/apache/camel/component/jooq/db/tables/daos/BookDao.java:[36,6]
cannot find symbol
[ERROR] symbol: class Autowired
[ERROR] location: class org.apache.camel.component.jooq.db.tables.daos.BookDao
[ERROR]
/home/apupier/git/camel/components/camel-jooq/target/generated-test-sources/jooq/org/apache/camel/component/jooq/db/tables/daos/BookStoreDao.java:[35,6]
cannot find symbol
[ERROR] symbol: class Autowired
[ERROR] location: class
org.apache.camel.component.jooq.db.tables.daos.BookStoreDao
[ERROR]
/home/apupier/git/camel/components/camel-jooq/target/generated-test-sources/jooq/org/apache/camel/component/jooq/db/tables/daos/BookToBookStoreDao.java:[35,6]
cannot find symbol
[ERROR] symbol: class Autowired
[ERROR] location: class
org.apache.camel.component.jooq.db.tables.daos.BookToBookStoreDao
[ERROR] -> [Help 2]
and we also have a lot of test failures. To improve that, i think we can:
* try to launch with the option locally first to not overcrowd the CI and
upload to develocity with a specific tag to be able to analyze easily the tests
which are not yet good for //. In fact, we could be able to first provide an
@Isolated on all of these ones in a first time, this should accelerate anyway
the global time.
* Then when able to have green build locally, try with a specific jenkins job,
separated from the others to avoid bringing too much instability
* Then enjoy it on the "normal" flow
Some ideas:
* to check also if we can do the pure compile in // with all threads and then
launch tests as of now (1 threads used although 22 are available on blarge
jenkins machines)
* see if it works and improve things for sonar analysis (40' on CI), this is a
separate command than the others
** no gain
[https://community.sonarsource.com/t/feature-suggestion-parallel-analysis-in-maven-sonarscanner/19876/11]
** tried locally with a local instance and it took 8'15" in both cases
** sonar scanner is unable to leverage multiple threads
* on GitHub there are already 2 phases but it is using mvnd; [it is taking
around 18 minutes|https://github.com/apache/camel/actions/runs/30412528538/
* evaluate with -T1C and more specific values, T1C might overload the Jenkins
nodes. To check.
* evaluate available memory too, most surefire tests are launch in forked
process with 2G of memory, if too many are launched in // it might exhaust.
Maybe can use lower in general and tweak with higher value only on modules
which requires it
* For deploy jobs, given that they are not running tests, see if it is
speeding things up, currently taking around 3 hours for Camel Core but
bottleneck might be the network more than the build itself
** locally with local repo:
*** 8'32" ( ./mvnw -T1C -U -B -e -fae -V -Dnoassembly
-Pdeploy,apache-snapshots -Dquickly clean deploy
-DaltDeploymentRepository=your_repo_id::default::file:/tmp/alt-repo-2 )
*** 18'33" (./mvnw -U -B -e -fae -V -Dnoassembly -Pdeploy,apache-snapshots
-Dquickly clean deploy
-DaltDeploymentRepository=your_repo_id::default::file:/tmp/alt-repo-2)
** on jenkins for Camel core:
*** 25' instead of usually 3 hours, previous record was 2h35 , see
https://ci-builds.apache.org/job/Camel/job/Camel%20Daily%20Snapshot%20Deploy/job/main/1494/
was:
Currently, the build on CI is launched using a single Core. Using multiple
cores should allow to drastically reduce the time.
Locally tests (22 cores, 32Gb RAM, similar to Jenkins large machines):
* -T1C for 642 modules, it took 40'
** command used: ./mvnw clean install -T1C -Dci.env.name=fakeCI-local -B -e
-fae -V -Dnoassembly -Dsurefire.rerunFailingTestsCount=2
-Dfailsafe.rerunFailingTestsCount=1 -Dchetype.test.skip
-Dmaven.test.failure.ignore=true -Dcheckstyle.skip=true install
-Dcamel.threads.virtual.enabled=false -Pcoverage
* without, it took 2h48minutes for the same 642 modules
** command used: ./mvnw clean install -Dci.env.name=fakeCI-local -B -e -fae -V
-Dnoassembly -Dsurefire.rerunFailingTestsCount=2
-Dfailsafe.rerunFailingTestsCount=1 -Dchetype.test.skip
-Dmaven.test.failure.ignore=true -Dcheckstyle.skip=true install
-Dcamel.threads.virtual.enabled=false -Pcoverage
Currently, the full compilation can fail when launched with -T1C
* [ERROR] Failed to execute goal
org.apache.maven.plugins:maven-remote-resources-plugin:3.3.0:process (default)
on project bundles-pom: Error loading supplemental data models: Could not find
resource 'target/legal/legal.xml'. -> [Help 1]
* this one also with single thread locally... [ERROR] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:3.15.0:compile (default-compile)
on project camel-jooq: Compilation failure: Compilation failure:
[ERROR]
/home/apupier/git/camel/components/camel-jooq/target/generated-test-sources/jooq/org/apache/camel/component/jooq/db/tables/daos/AuthorDao.java:[15,52]
package org.springframework.beans.factory.annotation does not exist
[ERROR]
/home/apupier/git/camel/components/camel-jooq/target/generated-test-sources/jooq/org/apache/camel/component/jooq/db/tables/daos/AuthorDao.java:[16,38]
package org.springframework.stereotype does not exist
[ERROR]
/home/apupier/git/camel/components/camel-jooq/target/generated-test-sources/jooq/org/apache/camel/component/jooq/db/tables/daos/AuthorDao.java:[23,2]
cannot find symbol
[ERROR] symbol: class Repository
[ERROR]
/home/apupier/git/camel/components/camel-jooq/target/generated-test-sources/jooq/org/apache/camel/component/jooq/db/tables/daos/BookDao.java:[15,52]
package org.springframework.beans.factory.annotation does not exist
[ERROR]
/home/apupier/git/camel/components/camel-jooq/target/generated-test-sources/jooq/org/apache/camel/component/jooq/db/tables/daos/BookDao.java:[16,38]
package org.springframework.stereotype does not exist
[ERROR]
/home/apupier/git/camel/components/camel-jooq/target/generated-test-sources/jooq/org/apache/camel/component/jooq/db/tables/daos/BookDao.java:[23,2]
cannot find symbol
[ERROR] symbol: class Repository
[ERROR]
/home/apupier/git/camel/components/camel-jooq/target/generated-test-sources/jooq/org/apache/camel/component/jooq/db/tables/daos/BookStoreDao.java:[14,52]
package org.springframework.beans.factory.annotation does not exist
[ERROR]
/home/apupier/git/camel/components/camel-jooq/target/generated-test-sources/jooq/org/apache/camel/component/jooq/db/tables/daos/BookStoreDao.java:[15,38]
package org.springframework.stereotype does not exist
[ERROR]
/home/apupier/git/camel/components/camel-jooq/target/generated-test-sources/jooq/org/apache/camel/component/jooq/db/tables/daos/BookStoreDao.java:[22,2]
cannot find symbol
[ERROR] symbol: class Repository
[ERROR]
/home/apupier/git/camel/components/camel-jooq/target/generated-test-sources/jooq/org/apache/camel/component/jooq/db/tables/daos/BookToBookStoreDao.java:[14,52]
package org.springframework.beans.factory.annotation does not exist
[ERROR]
/home/apupier/git/camel/components/camel-jooq/target/generated-test-sources/jooq/org/apache/camel/component/jooq/db/tables/daos/BookToBookStoreDao.java:[15,38]
package org.springframework.stereotype does not exist
[ERROR]
/home/apupier/git/camel/components/camel-jooq/target/generated-test-sources/jooq/org/apache/camel/component/jooq/db/tables/daos/BookToBookStoreDao.java:[22,2]
cannot find symbol
[ERROR] symbol: class Repository
[ERROR]
/home/apupier/git/camel/components/camel-jooq/target/generated-test-sources/jooq/org/apache/camel/component/jooq/db/tables/daos/AuthorDao.java:[36,6]
cannot find symbol
[ERROR] symbol: class Autowired
[ERROR] location: class org.apache.camel.component.jooq.db.tables.daos.AuthorDao
[ERROR]
/home/apupier/git/camel/components/camel-jooq/target/generated-test-sources/jooq/org/apache/camel/component/jooq/db/tables/daos/BookDao.java:[36,6]
cannot find symbol
[ERROR] symbol: class Autowired
[ERROR] location: class org.apache.camel.component.jooq.db.tables.daos.BookDao
[ERROR]
/home/apupier/git/camel/components/camel-jooq/target/generated-test-sources/jooq/org/apache/camel/component/jooq/db/tables/daos/BookStoreDao.java:[35,6]
cannot find symbol
[ERROR] symbol: class Autowired
[ERROR] location: class
org.apache.camel.component.jooq.db.tables.daos.BookStoreDao
[ERROR]
/home/apupier/git/camel/components/camel-jooq/target/generated-test-sources/jooq/org/apache/camel/component/jooq/db/tables/daos/BookToBookStoreDao.java:[35,6]
cannot find symbol
[ERROR] symbol: class Autowired
[ERROR] location: class
org.apache.camel.component.jooq.db.tables.daos.BookToBookStoreDao
[ERROR] -> [Help 2]
and we also have a lot of test failures. To improve that, i think we can:
* try to launch with the option locally first to not overcrowd the CI and
upload to develocity with a specific tag to be able to analyze easily the tests
which are not yet good for //. In fact, we could be able to first provide an
@Isolated on all of these ones in a first time, this should accelerate anyway
the global time.
* Then when able to have green build locally, try with a specific jenkins job,
separated from the others to avoid bringing too much instability
* Then enjoy it on the "normal" flow
Some ideas:
* to check also if we can do the pure compile in // with all threads and then
launch tests as of now (1 threads used although 22 are available on blarge
jenkins machines)
* see if it works and improve things for sonar analysis (40' on CI), this is a
separate command than the others
** no gain
[https://community.sonarsource.com/t/feature-suggestion-parallel-analysis-in-maven-sonarscanner/19876/11]
** tried locally with a local instance and it took 8'15" in both cases
** sonar scanner is unable to leverage multiple threads
* on GitHub there are already 2 phases but it is using mvnd; [it is taking
around 18 minutes|https://github.com/apache/camel/actions/runs/30412528538/
* evaluate with -T1C and more specific values, T1C might overload the Jenkins
nodes. To check.
* evaluate available memory too, most surefire tests are launch in forked
process with 2G of memory, if too many are launched in // it might exhaust.
Maybe can use lower in general and tweak with higher value only on modules
which requires it
* For deploy jobs, given that they are not running tests, see if it is
speeding things up, currently taking around 3 hours for Camel Core but
bottleneck might be the network more than the build itself
> Support Maven build and test using multiple cores
> -------------------------------------------------
>
> Key: CAMEL-24318
> URL: https://issues.apache.org/jira/browse/CAMEL-24318
> Project: Camel
> Issue Type: Task
> Components: build system
> Affects Versions: 4.21.0
> Reporter: Aurélien Pupier
> Priority: Major
>
> Currently, the build on CI is launched using a single Core. Using multiple
> cores should allow to drastically reduce the time.
> Locally tests (22 cores, 32Gb RAM, similar to Jenkins large machines):
> * -T1C for 642 modules, it took 40'
> ** command used: ./mvnw clean install -T1C -Dci.env.name=fakeCI-local -B -e
> -fae -V -Dnoassembly -Dsurefire.rerunFailingTestsCount=2
> -Dfailsafe.rerunFailingTestsCount=1 -Dchetype.test.skip
> -Dmaven.test.failure.ignore=true -Dcheckstyle.skip=true install
> -Dcamel.threads.virtual.enabled=false -Pcoverage
> * without, it took 2h48minutes for the same 642 modules
> ** command used: ./mvnw clean install -Dci.env.name=fakeCI-local -B -e -fae
> -V -Dnoassembly -Dsurefire.rerunFailingTestsCount=2
> -Dfailsafe.rerunFailingTestsCount=1 -Dchetype.test.skip
> -Dmaven.test.failure.ignore=true -Dcheckstyle.skip=true install
> -Dcamel.threads.virtual.enabled=false -Pcoverage
>
> Currently, the full compilation can fail when launched with -T1C
> * [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-remote-resources-plugin:3.3.0:process
> (default) on project bundles-pom: Error loading supplemental data models:
> Could not find resource 'target/legal/legal.xml'. -> [Help 1]
> * this one also with single thread locally... [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-compiler-plugin:3.15.0:compile
> (default-compile) on project camel-jooq: Compilation failure: Compilation
> failure:
> [ERROR]
> /home/apupier/git/camel/components/camel-jooq/target/generated-test-sources/jooq/org/apache/camel/component/jooq/db/tables/daos/AuthorDao.java:[15,52]
> package org.springframework.beans.factory.annotation does not exist
> [ERROR]
> /home/apupier/git/camel/components/camel-jooq/target/generated-test-sources/jooq/org/apache/camel/component/jooq/db/tables/daos/AuthorDao.java:[16,38]
> package org.springframework.stereotype does not exist
> [ERROR]
> /home/apupier/git/camel/components/camel-jooq/target/generated-test-sources/jooq/org/apache/camel/component/jooq/db/tables/daos/AuthorDao.java:[23,2]
> cannot find symbol
> [ERROR] symbol: class Repository
> [ERROR]
> /home/apupier/git/camel/components/camel-jooq/target/generated-test-sources/jooq/org/apache/camel/component/jooq/db/tables/daos/BookDao.java:[15,52]
> package org.springframework.beans.factory.annotation does not exist
> [ERROR]
> /home/apupier/git/camel/components/camel-jooq/target/generated-test-sources/jooq/org/apache/camel/component/jooq/db/tables/daos/BookDao.java:[16,38]
> package org.springframework.stereotype does not exist
> [ERROR]
> /home/apupier/git/camel/components/camel-jooq/target/generated-test-sources/jooq/org/apache/camel/component/jooq/db/tables/daos/BookDao.java:[23,2]
> cannot find symbol
> [ERROR] symbol: class Repository
> [ERROR]
> /home/apupier/git/camel/components/camel-jooq/target/generated-test-sources/jooq/org/apache/camel/component/jooq/db/tables/daos/BookStoreDao.java:[14,52]
> package org.springframework.beans.factory.annotation does not exist
> [ERROR]
> /home/apupier/git/camel/components/camel-jooq/target/generated-test-sources/jooq/org/apache/camel/component/jooq/db/tables/daos/BookStoreDao.java:[15,38]
> package org.springframework.stereotype does not exist
> [ERROR]
> /home/apupier/git/camel/components/camel-jooq/target/generated-test-sources/jooq/org/apache/camel/component/jooq/db/tables/daos/BookStoreDao.java:[22,2]
> cannot find symbol
> [ERROR] symbol: class Repository
> [ERROR]
> /home/apupier/git/camel/components/camel-jooq/target/generated-test-sources/jooq/org/apache/camel/component/jooq/db/tables/daos/BookToBookStoreDao.java:[14,52]
> package org.springframework.beans.factory.annotation does not exist
> [ERROR]
> /home/apupier/git/camel/components/camel-jooq/target/generated-test-sources/jooq/org/apache/camel/component/jooq/db/tables/daos/BookToBookStoreDao.java:[15,38]
> package org.springframework.stereotype does not exist
> [ERROR]
> /home/apupier/git/camel/components/camel-jooq/target/generated-test-sources/jooq/org/apache/camel/component/jooq/db/tables/daos/BookToBookStoreDao.java:[22,2]
> cannot find symbol
> [ERROR] symbol: class Repository
> [ERROR]
> /home/apupier/git/camel/components/camel-jooq/target/generated-test-sources/jooq/org/apache/camel/component/jooq/db/tables/daos/AuthorDao.java:[36,6]
> cannot find symbol
> [ERROR] symbol: class Autowired
> [ERROR] location: class
> org.apache.camel.component.jooq.db.tables.daos.AuthorDao
> [ERROR]
> /home/apupier/git/camel/components/camel-jooq/target/generated-test-sources/jooq/org/apache/camel/component/jooq/db/tables/daos/BookDao.java:[36,6]
> cannot find symbol
> [ERROR] symbol: class Autowired
> [ERROR] location: class org.apache.camel.component.jooq.db.tables.daos.BookDao
> [ERROR]
> /home/apupier/git/camel/components/camel-jooq/target/generated-test-sources/jooq/org/apache/camel/component/jooq/db/tables/daos/BookStoreDao.java:[35,6]
> cannot find symbol
> [ERROR] symbol: class Autowired
> [ERROR] location: class
> org.apache.camel.component.jooq.db.tables.daos.BookStoreDao
> [ERROR]
> /home/apupier/git/camel/components/camel-jooq/target/generated-test-sources/jooq/org/apache/camel/component/jooq/db/tables/daos/BookToBookStoreDao.java:[35,6]
> cannot find symbol
> [ERROR] symbol: class Autowired
> [ERROR] location: class
> org.apache.camel.component.jooq.db.tables.daos.BookToBookStoreDao
> [ERROR] -> [Help 2]
> and we also have a lot of test failures. To improve that, i think we can:
> * try to launch with the option locally first to not overcrowd the CI and
> upload to develocity with a specific tag to be able to analyze easily the
> tests which are not yet good for //. In fact, we could be able to first
> provide an @Isolated on all of these ones in a first time, this should
> accelerate anyway the global time.
> * Then when able to have green build locally, try with a specific jenkins
> job, separated from the others to avoid bringing too much instability
> * Then enjoy it on the "normal" flow
> Some ideas:
> * to check also if we can do the pure compile in // with all threads and
> then launch tests as of now (1 threads used although 22 are available on
> blarge jenkins machines)
> * see if it works and improve things for sonar analysis (40' on CI), this is
> a separate command than the others
> ** no gain
> [https://community.sonarsource.com/t/feature-suggestion-parallel-analysis-in-maven-sonarscanner/19876/11]
> ** tried locally with a local instance and it took 8'15" in both cases
> ** sonar scanner is unable to leverage multiple threads
> * on GitHub there are already 2 phases but it is using mvnd; [it is taking
> around 18 minutes|https://github.com/apache/camel/actions/runs/30412528538/
> * evaluate with -T1C and more specific values, T1C might overload the
> Jenkins nodes. To check.
> * evaluate available memory too, most surefire tests are launch in forked
> process with 2G of memory, if too many are launched in // it might exhaust.
> Maybe can use lower in general and tweak with higher value only on modules
> which requires it
> * For deploy jobs, given that they are not running tests, see if it is
> speeding things up, currently taking around 3 hours for Camel Core but
> bottleneck might be the network more than the build itself
> ** locally with local repo:
> *** 8'32" ( ./mvnw -T1C -U -B -e -fae -V -Dnoassembly
> -Pdeploy,apache-snapshots -Dquickly clean deploy
> -DaltDeploymentRepository=your_repo_id::default::file:/tmp/alt-repo-2 )
> *** 18'33" (./mvnw -U -B -e -fae -V -Dnoassembly -Pdeploy,apache-snapshots
> -Dquickly clean deploy
> -DaltDeploymentRepository=your_repo_id::default::file:/tmp/alt-repo-2)
> ** on jenkins for Camel core:
> *** 25' instead of usually 3 hours, previous record was 2h35 , see
> https://ci-builds.apache.org/job/Camel/job/Camel%20Daily%20Snapshot%20Deploy/job/main/1494/
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
