[
https://issues.apache.org/jira/browse/CAMEL-24290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aurélien Pupier updated CAMEL-24290:
------------------------------------
Description:
Our 2 CI systems (Jenkins and GitHub actions) are already under heavy loads and
the planned future will surely increase the load and the need for a more stable
CI.
Currently Jenkins is taking between 4 and 5 hours when things are going well.
Can go up to 20 hours when all streams in //
On GitHub, it happens regularly that the jobs are queued for several hours.
We have few full green build per week on Jenkins. (We are at one or two per
year last year, so already a lot better but not enough to really sustain new
workloads efficiently)
There are several factors for increased, and potential new increase:
* Increased number PRs with more review, so more cycle
* Increased number of backports
* Increased cadence of dependencies release
* Longer LTS ? , more streams to maintain?
* Shared GitHub runners with all Apache projects (and several of them are
facing the same workload increase)
Several areas identified which worth at least some investigations:
* stability of Workflows run: it gives more confidence to the build, avoids
the need to rerun to check that it is "just" flaky or an infra hiccups
** develocity help tracking flaky tests (especially flaky tests causing
failure)
** one last test is failing very often:
https://issues.apache.org/jira/browse/CAMEL-24284
** several failing less often (every months)
** and also several flaky during the run but passing in less than 3 attempts,
slowing down the build
** one workspace leak to fix https://issues.apache.org/jira/browse/CAMEL-23795
* modification of test matrixes
** testing a single JDK/OS permanently and push to weekly/nightly others on
jenkins?
** Reduce test matrix on maintenance stream?
* Optimize usage of available resources
** see if we can build on smaller instances of Jenkins, for Linux it will
allow to move up from 4 to ~20 instances available.
** use several threads (but requires a far better stability and robustness of
tests and ensure no port conflict)
*** to check also if we can do the pure compile in // with all threads and
then launch tests as of now (1 threads used although 22 are available on big
machines)
**** on GitHub there are already 2 phases but it is using mvnd; [it is taking
around 18
minutes|https://github.com/apache/camel/actions/runs/30412528538/job/90451766586#step:9:1]
**** on jenkins it means to split in different phase
*** on Jenkins, I think we could attempt to provide a specific job for this
experiment with specific develocity tag so that can filter out/in the results
easily
** also seems 3.5G of Memory is used for the build, maybe we could use more
when in //
*** 32G in total on large jenkins machine
*** but [develocity seems to report only 1.7
max|https://develocity.apache.org/s/kuigqxgdpicsy/performance/build] is used
currently
*** maybe an indication that we could now run on smaller Jenkins node as this
Memory constraint seems to date from 2018 and it has not changed, good chance
that jenkins nodes size has been improved since that time
* several s390x and ppc64le in //, currently [3
ppc64le|https://jenkins-ccos.apache.org/view/Shared%20-%20ppc64le%20nodes/]
and [6 s390x|https://jenkins-ccos.apache.org/view/Shared%20-%20s390x%20nodes/]
nodes are referenced as shared node but a single one is available at a time
* improve performance of the product/test code itself maybe the trickier one
but also the most beneficial for our end users
* use of Develocity build cache (not a big fan of not rebuilding but could be
something to reuse in some configurations)
* use of Develocity test selection
* improve scalpel to reduce even more the scope of PR tests
* improve [fetch based branch for scalpel change detection on Jenkins which is
taking almost 5
minutes|https://github.com/apache/camel/actions/runs/30429114691/job/90501998254#step:3:1]
* gating PR builds?
* see if budget to provide more github runners and/or jenkins nodes
* check for performance bottleneck in the build and test process (develocity
build scan performance can give some ideas)
was:
Our 2 CI systems (Jenkins and GitHub actions) are already under heavy loads and
the planned future will surely increase the load and the need for a more stable
CI.
Currently Jenkins is taking between 4 and 5 hours when things are going well.
Can go up to 20 hours when all streams in //
On GitHub, it happens regularly that the jobs are queued for several hours.
We have few full green build per week on Jenkins. (We are at one or two per
year last year, so already a lot better but not enough to really sustain new
workloads efficiently)
There are several factors for increased, and potential new increase:
* Increased number PRs with more review, so more cycle
* Increased number of backports
* Increased cadence of dependencies release
* Longer LTS ? , more streams to maintain?
* Shared GitHub runners with all Apache projects (and several of them are
facing the same workload increase)
Several areas identified which worth at least some investigations:
* stability of Workflows run: it gives more confidence to the build, avoids
the need to rerun to check that it is "just" flaky or an infra hiccups
** develocity help tracking flaky tests (especially flaky tests causing
failure)
** one last test is failing very often:
https://issues.apache.org/jira/browse/CAMEL-24284
** several failing less often (every months)
** and also several flaky during the run but passing in less than 3 attempts,
slowing down the build
** one workspace leak to fix
* modification of test matrixes
** testing a single JDK/OS permanently and push to weekly/nightly others on
jenkins?
** Reduce test matrix on maintenance stream?
* Optimize usage of available resources
** see if we can build on smaller instances of Jenkins, for Linux it will
allow to move up from 4 to ~20 instances available.
** use several threads (but requires a far better stability and robustness of
tests and ensure no port conflict)
*** to check also if we can do the pure compile in // with all threads and
then launch tests as of now (1 threads used although 22 are available on big
machines)
**** on GitHub there are already 2 phases but it is using mvnd; [it is taking
around 18
minutes|https://github.com/apache/camel/actions/runs/30412528538/job/90451766586#step:9:1]
**** on jenkins it means to split in different phase
*** on Jenkins, I think we could attempt to provide a specific job for this
experiment with specific develocity tag so that can filter out/in the results
easily
** also seems 3.5G of Memory is used for the build, maybe we could use more
when in //
*** 32G in total on large jenkins machine
*** but [develocity seems to report only 1.7
max|https://develocity.apache.org/s/kuigqxgdpicsy/performance/build] is used
currently
*** maybe an indication that we could now run on smaller Jenkins node as this
Memory constraint seems to date from 2018 and it has not changed, good chance
that jenkins nodes size has been improved since that time
* several s390x and ppc64le in //, currently [3
ppc64le|https://jenkins-ccos.apache.org/view/Shared%20-%20ppc64le%20nodes/]
and [6 s390x|https://jenkins-ccos.apache.org/view/Shared%20-%20s390x%20nodes/]
nodes are referenced as shared node but a single one is available at a time
* improve performance of the product/test code itself maybe the trickier one
but also the most beneficial for our end users
* use of Develocity build cache (not a big fan of not rebuilding but could be
something to reuse in some configurations)
* use of Develocity test selection
* improve scalpel to reduce even more the scope of PR tests
* improve [fetch based branch for scalpel change detection on Jenkins which is
taking almost 5
minutes|https://github.com/apache/camel/actions/runs/30429114691/job/90501998254#step:3:1]
* gating PR builds?
* see if budget to provide more github runners and/or jenkins nodes
* check for performance bottleneck in the build and test process (develocity
build scan performance can give some ideas)
> Improve Continuous Integration to support larger workload
> ---------------------------------------------------------
>
> Key: CAMEL-24290
> URL: https://issues.apache.org/jira/browse/CAMEL-24290
> Project: Camel
> Issue Type: Task
> Components: build system
> Affects Versions: 4.21.0
> Reporter: Aurélien Pupier
> Priority: Major
>
> Our 2 CI systems (Jenkins and GitHub actions) are already under heavy loads
> and the planned future will surely increase the load and the need for a more
> stable CI.
> Currently Jenkins is taking between 4 and 5 hours when things are going well.
> Can go up to 20 hours when all streams in //
> On GitHub, it happens regularly that the jobs are queued for several hours.
> We have few full green build per week on Jenkins. (We are at one or two per
> year last year, so already a lot better but not enough to really sustain new
> workloads efficiently)
>
> There are several factors for increased, and potential new increase:
> * Increased number PRs with more review, so more cycle
> * Increased number of backports
> * Increased cadence of dependencies release
> * Longer LTS ? , more streams to maintain?
> * Shared GitHub runners with all Apache projects (and several of them are
> facing the same workload increase)
> Several areas identified which worth at least some investigations:
> * stability of Workflows run: it gives more confidence to the build, avoids
> the need to rerun to check that it is "just" flaky or an infra hiccups
> ** develocity help tracking flaky tests (especially flaky tests causing
> failure)
> ** one last test is failing very often:
> https://issues.apache.org/jira/browse/CAMEL-24284
> ** several failing less often (every months)
> ** and also several flaky during the run but passing in less than 3
> attempts, slowing down the build
> ** one workspace leak to fix
> https://issues.apache.org/jira/browse/CAMEL-23795
> * modification of test matrixes
> ** testing a single JDK/OS permanently and push to weekly/nightly others on
> jenkins?
> ** Reduce test matrix on maintenance stream?
> * Optimize usage of available resources
> ** see if we can build on smaller instances of Jenkins, for Linux it will
> allow to move up from 4 to ~20 instances available.
> ** use several threads (but requires a far better stability and robustness
> of tests and ensure no port conflict)
> *** to check also if we can do the pure compile in // with all threads and
> then launch tests as of now (1 threads used although 22 are available on big
> machines)
> **** on GitHub there are already 2 phases but it is using mvnd; [it is
> taking around 18
> minutes|https://github.com/apache/camel/actions/runs/30412528538/job/90451766586#step:9:1]
> **** on jenkins it means to split in different phase
> *** on Jenkins, I think we could attempt to provide a specific job for this
> experiment with specific develocity tag so that can filter out/in the results
> easily
> ** also seems 3.5G of Memory is used for the build, maybe we could use more
> when in //
> *** 32G in total on large jenkins machine
> *** but [develocity seems to report only 1.7
> max|https://develocity.apache.org/s/kuigqxgdpicsy/performance/build] is used
> currently
> *** maybe an indication that we could now run on smaller Jenkins node as
> this Memory constraint seems to date from 2018 and it has not changed, good
> chance that jenkins nodes size has been improved since that time
> * several s390x and ppc64le in //, currently [3
> ppc64le|https://jenkins-ccos.apache.org/view/Shared%20-%20ppc64le%20nodes/]
> and [6
> s390x|https://jenkins-ccos.apache.org/view/Shared%20-%20s390x%20nodes/] nodes
> are referenced as shared node but a single one is available at a time
> * improve performance of the product/test code itself maybe the trickier one
> but also the most beneficial for our end users
> * use of Develocity build cache (not a big fan of not rebuilding but could
> be something to reuse in some configurations)
> * use of Develocity test selection
> * improve scalpel to reduce even more the scope of PR tests
> * improve [fetch based branch for scalpel change detection on Jenkins which
> is taking almost 5
> minutes|https://github.com/apache/camel/actions/runs/30429114691/job/90501998254#step:3:1]
> * gating PR builds?
> * see if budget to provide more github runners and/or jenkins nodes
> * check for performance bottleneck in the build and test process (develocity
> build scan performance can give some ideas)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
