[
https://issues.apache.org/jira/browse/CAMEL-24290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18142127#comment-18142127
]
Aurélien Pupier edited comment on CAMEL-24290 at 11/10/26, 4:13 PM:
--------------------------------------------------------------------
_Analysis generated by Bob (IBM Bob, AI assistant) on behalf of apupier.
Figures are derived from the Develocity build scan timeline data and should be
verified before acting on them._
h3. Critical path analysis of build scan vq6u4jcg4egze
Scan: https://develocity.apache.org/s/vq6u4jcg4egze/timeline
Total build duration: 2278.6 s (the previous scan eb33cs73xhkq4 took 2303.4 s).
h4. Method
* Per-module span = first goal start to last goal end, taken from
{{/scan-data/maven/<scan>/timeline}}.
* Dependency edges inferred from {{pom.xml}} files (parent, dependencies and
plugins, with parent inheritance).
* Critical path = start at the last-finishing module and repeatedly step back
to the dependency that finished last before the module started. Gap between
consecutive path modules is at most 2.6 s, and the path has 51 modules.
* CPU = mean of the machine-wide {{allProcessesCpu}} metric (about 1 s
sampling) over each module's span. "% time <=60% CPU" is the share of the
module's time at or below 60% load. Idle-seconds = duration x (100 - mean CPU%).
h4. Critical path modules of at least 10 s (path order)
||Module||Start (s)||Duration||CPU %||% time <=60% CPU||Idle-s||Longest goal||
|Util|36|10.4 s|42|70|6|surefire:test 6 s|
|Support|54|11.6 s|40|83|7|surefire:test 3 s|
|XML IO|89|23.6 s|66|43|8|jaxb2:schemagen 8 s|
|[*Core*|https://issues.apache.org/jira/browse/CAMEL-25893]|117|*243.3
s*|23|91|*188*|surefire:test 213 s|
|[Management|https://issues.apache.org/jira/browse/CAMEL-25965]|360|76.8
s|29|99|55|surefire:test 67 s|
|Test :: JUnit6|437|17.3 s|52|53|8|surefire:test 16 s|
|Spring XML|828|67.8 s|74|25|17|surefire:test 43 s|
|Test :: Spring :: JUnit6|896|32.7 s|61|42|13|surefire:test 31 s|
|*HTTP*|929|*315.5 s*|98|0|5|surefire:test 287 s|
|Jetty|1244|54.4 s|92|0|4|surefire:test 45 s|
|*Netty HTTP*|1299|*309.5 s*|89|12|34|surefire:test 304 s|
|Telegram|1615|64.8 s|45|85|36|surefire:test 60 s|
|AI :: MCP Server|1680|63.6 s|41|84|38|failsafe:integration-test 34 s|
|Catalog :: Camel Catalog|1746|27.8 s|25|100|21|surefire:test 22 s|
|YAML DSL|1782|23.3 s|54|57|11|surefire:test 15 s|
|YAML DSL :: Validator|1806|21.9 s|65|36|8|surefire:test 19 s|
|[*JBang ::
Core*|https://issues.apache.org/jira/browse/CAMEL-26818]|1830|*304.1
s*|27|92|*222*|surefire:test serial-test 198 s|
|[JBang :: Plugin ::
TUI|https://issues.apache.org/jira/browse/CAMEL-25966]|2134|69.2
s|37|71|43|surefire:test 42 s|
|Launcher|2203|69.9 s|32|90|48|surefire:test 54 s|
Not listed: 16 modules between AI :: Hugging Face and Zendesk (about 454 s to
828 s, about 560 s in total). All run at 94-99% CPU with 0-5% of their time at
or below 60% load.
h4. Long modules that also have CPU available (ranked by idle-seconds)
# *JBang :: Core*: 304 s at 27% CPU, 222 idle-s. The {{serial-test}} execution
alone is 198 s.
# *Core*: 243 s at 23% CPU, 188 idle-s.
# *Management*: 77 s at 29% CPU, 55 idle-s.
# *Launcher*: 70 s at 32% CPU, 48 idle-s.
# *JBang :: Plugin :: TUI*: 69 s at 37% CPU, 43 idle-s.
# AI :: MCP Server, Telegram and Catalog: about 21-38 idle-s each.
h4. Long modules where the machine is saturated
* *HTTP* (315 s) and *Netty HTTP* (310 s, 12% of the time at or below 60% CPU),
plus the 16 modules in the 454-828 s stretch. Parallelising their tests is
unlikely to help, since other modules keep the machine near 100%.
h4. Change since the previous scan (eb33cs73xhkq4)
* Spring XML dropped from 116.6 s to 67.8 s ({{surefire:test}} 90 s to 43 s).
* Core dropped from 265.9 s to 243.3 s.
* JBang :: Core went up from 281.1 s to 304.1 s ({{serial-test}} 179 s to 198
s).
* Total build time dropped by about 25 s.
h4. Caveats
* Dependency edges are inferred from pom files, not reported by Develocity. The
spine Core -> HTTP -> Netty HTTP -> JBang :: Core is well supported (gaps of
0.1 s or less). The stretch between about 454 s and 828 s depends on Maven
{{-T1C}} scheduling and is not a fixed dependency chain, so it differs between
scans.
* CPU is machine-wide (all processes on the agent), not per module, and sampled
at about 1 s.
* The data does not show why a module leaves CPU idle (serial tests, sleeps or
timeouts). That needs a look at the surefire configuration and per-test
durations.
was (Author: apupier):
_Analysis generated by Bob (IBM Bob, AI assistant) on behalf of apupier.
Figures are derived from the Develocity build scan timeline data and should be
verified before acting on them._
h3. Critical path analysis of build scan vq6u4jcg4egze
Scan: https://develocity.apache.org/s/vq6u4jcg4egze/timeline
Total build duration: 2278.6 s (the previous scan eb33cs73xhkq4 took 2303.4 s).
h4. Method
* Per-module span = first goal start to last goal end, taken from
{{/scan-data/maven/<scan>/timeline}}.
* Dependency edges inferred from {{pom.xml}} files (parent, dependencies and
plugins, with parent inheritance).
* Critical path = start at the last-finishing module and repeatedly step back
to the dependency that finished last before the module started. Gap between
consecutive path modules is at most 2.6 s, and the path has 51 modules.
* CPU = mean of the machine-wide {{allProcessesCpu}} metric (about 1 s
sampling) over each module's span. "% time <=60% CPU" is the share of the
module's time at or below 60% load. Idle-seconds = duration x (100 - mean CPU%).
h4. Critical path modules of at least 10 s (path order)
||Module||Start (s)||Duration||CPU %||% time <=60% CPU||Idle-s||Longest goal||
|Util|36|10.4 s|42|70|6|surefire:test 6 s|
|Support|54|11.6 s|40|83|7|surefire:test 3 s|
|XML IO|89|23.6 s|66|43|8|jaxb2:schemagen 8 s|
|[*Core*|https://issues.apache.org/jira/browse/CAMEL-25893]|117|*243.3
s*|23|91|*188*|surefire:test 213 s|
|[Management|https://issues.apache.org/jira/browse/CAMEL-25965]|360|76.8
s|29|99|55|surefire:test 67 s|
|Test :: JUnit6|437|17.3 s|52|53|8|surefire:test 16 s|
|Spring XML|828|67.8 s|74|25|17|surefire:test 43 s|
|Test :: Spring :: JUnit6|896|32.7 s|61|42|13|surefire:test 31 s|
|*HTTP*|929|*315.5 s*|98|0|5|surefire:test 287 s|
|Jetty|1244|54.4 s|92|0|4|surefire:test 45 s|
|*Netty HTTP*|1299|*309.5 s*|89|12|34|surefire:test 304 s|
|Telegram|1615|64.8 s|45|85|36|surefire:test 60 s|
|AI :: MCP Server|1680|63.6 s|41|84|38|failsafe:integration-test 34 s|
|Catalog :: Camel Catalog|1746|27.8 s|25|100|21|surefire:test 22 s|
|YAML DSL|1782|23.3 s|54|57|11|surefire:test 15 s|
|YAML DSL :: Validator|1806|21.9 s|65|36|8|surefire:test 19 s|
|*JBang :: Core*|1830|*304.1 s*|27|92|*222*|surefire:test serial-test 198 s|
|[JBang :: Plugin ::
TUI|https://issues.apache.org/jira/browse/CAMEL-25966]|2134|69.2
s|37|71|43|surefire:test 42 s|
|Launcher|2203|69.9 s|32|90|48|surefire:test 54 s|
Not listed: 16 modules between AI :: Hugging Face and Zendesk (about 454 s to
828 s, about 560 s in total). All run at 94-99% CPU with 0-5% of their time at
or below 60% load.
h4. Long modules that also have CPU available (ranked by idle-seconds)
# *JBang :: Core*: 304 s at 27% CPU, 222 idle-s. The {{serial-test}} execution
alone is 198 s.
# *Core*: 243 s at 23% CPU, 188 idle-s.
# *Management*: 77 s at 29% CPU, 55 idle-s.
# *Launcher*: 70 s at 32% CPU, 48 idle-s.
# *JBang :: Plugin :: TUI*: 69 s at 37% CPU, 43 idle-s.
# AI :: MCP Server, Telegram and Catalog: about 21-38 idle-s each.
h4. Long modules where the machine is saturated
* *HTTP* (315 s) and *Netty HTTP* (310 s, 12% of the time at or below 60% CPU),
plus the 16 modules in the 454-828 s stretch. Parallelising their tests is
unlikely to help, since other modules keep the machine near 100%.
h4. Change since the previous scan (eb33cs73xhkq4)
* Spring XML dropped from 116.6 s to 67.8 s ({{surefire:test}} 90 s to 43 s).
* Core dropped from 265.9 s to 243.3 s.
* JBang :: Core went up from 281.1 s to 304.1 s ({{serial-test}} 179 s to 198
s).
* Total build time dropped by about 25 s.
h4. Caveats
* Dependency edges are inferred from pom files, not reported by Develocity. The
spine Core -> HTTP -> Netty HTTP -> JBang :: Core is well supported (gaps of
0.1 s or less). The stretch between about 454 s and 828 s depends on Maven
{{-T1C}} scheduling and is not a fixed dependency chain, so it differs between
scans.
* CPU is machine-wide (all processes on the agent), not per module, and sampled
at about 1 s.
* The data does not show why a module leaves CPU idle (serial tests, sleeps or
timeouts). That needs a look at the surefire configuration and per-test
durations.
> Improve Continuous Integration to support larger workload
> ---------------------------------------------------------
>
> Key: CAMEL-24290
> URL: https://issues.apache.org/jira/browse/CAMEL-24290
> Project: Camel
> Issue Type: Task
> Components: build system
> Affects Versions: 4.21.0
> Reporter: Aurélien Pupier
> Priority: Major
>
> Our 2 CI systems (Jenkins and GitHub actions) are already under heavy loads
> and the planned future will surely increase the load and the need for a more
> stable CI.
> Currently Jenkins is taking between 4 and 5 hours when things are going well.
> Can go up to 20 hours when all streams in //
> On GitHub, it happens regularly that the jobs are queued for several hours.
> We have few full green build per week on Jenkins. (We are at one or two per
> year last year, so already a lot better but not enough to really sustain new
> workloads efficiently).
> Having a CI failing only for real issues will also help all committers to
> react faster
>
> There are several factors for increased, and potential new increase:
> * Increased number PRs with more review, so more cycle
> * Increased number of backports
> * Increased cadence of dependencies release
> * Longer LTS ? , more streams to maintain?
> * Shared GitHub runners with all Apache projects (and several of them are
> facing the same workload increase)
> Several areas identified which worth at least some investigations:
> * stability of Workflows run: it gives more confidence to the build, avoids
> the need to rerun to check that it is "just" flaky or an infra hiccups
> ** develocity help tracking flaky tests (especially flaky tests causing
> failure)
> ** -one last test is failing very often: SjmsConnectionRecoveryTest-
> -[https://issues.apache.org/jira/browse/CAMEL-24284-] DONE no failure since
> 1st September
> ** several failing less often (every months)
> *** useful Develocity dashboard
> [https://develocity.apache.org/scans/tests?search.query=project:camel%20tag:main%20-maven.environment.jvmVersion:*26*&search.relativeStartTime=P28D]
> **** beware of tests which failed for good reasons (it caught regressions)
> **** beware of all the tests which were already fixed (there was a huge
> session of fix, so maybe need to wait a month to have a clearer view and/or
> play with the the timeframe)
> **** I filtered on main and excluded JDK 26 for now. Because i have JDK 26
> has tests which are failing for an identified bug and a lot of stability
> improvements were done on main only
> **** based on previous warning, the current curated list is:
> ***** MasterEndpointIT
> [https://develocity.apache.org/scans/tests?search.query=project:camel%20tag:main%20-maven.environment.jvmVersion:*26*&search.timeZoneId=Europe%2FParis&tests.container=org.apache.camel.component.zookeepermaster.MasterEndpointIT&tests.test=testEndpoint]
> ***** MongoDB:
> [MongoDbBulkWriteOperationsIT|https://develocity.apache.org/scans/tests?search.query=project:camel%20tag:main%20-maven.environment.jvmVersion:*26*&search.timeZoneId=Europe%2FParis&tests.container=org.apache.camel.component.mongodb.integration.MongoDbBulkWriteOperationIT],
>
> [MongoDbCredentialsFromUriConnectionIT|https://develocity.apache.org/scans/tests?search.query=project:camel%20tag:main%20-maven.environment.jvmVersion:*26*&search.timeZoneId=Europe%2FParis&tests.container=org.apache.camel.component.mongodb.integration.MongoDbCredentialsFromUriConnectionIT]
> and
> [MongoDbFindOneAndReplaceOperationIT|https://develocity.apache.org/scans/tests?search.query=project:camel%20tag:main%20-maven.environment.jvmVersion:*26*&search.timeZoneId=Europe%2FParis&tests.container=org.apache.camel.component.mongodb.integration.MongoDbFindOneAndReplaceOperationIT]
> *****
> [SpringFileAntPathMatcherRemoteFileFilterTest|https://develocity.apache.org/scans/tests?search.query=project:camel%20tag:main%20-maven.environment.jvmVersion:*26*&search.timeZoneId=Europe%2FParis&tests.container=org.apache.camel.itest.ftp.SpringFileAntPathMatcherRemoteFileFilterTest]
> CAMEL-24618
> *****
> [CassandraIdempotentRespotoryIT|https://develocity.apache.org/scans/tests?search.query=project:camel%20tag:main%20-maven.environment.jvmVersion:*26*&search.timeZoneId=Europe%2FParis&tests.container=org.apache.camel.processor.idempotent.cassandra.CassandraIdempotentRepositoryIT]
> ***** and certainly a few others currently buried in the list of already
> fixed ones
> ** and also several flaky during the run but passing in less than 3
> attempts, slowing down the build
> ***
> [https://develocity.apache.org/scans/tests?search.query=project:camel%20tag:main&search.relativeStartTime=P28D&search.timeZoneId=Europe%2FParis&tests.sortField=FLAKY]
> ***
> ** -one Jenkins workspace leak to fix due to some jbang tests- CAMEL-23795
> DONE
> * modification of test matrixes
> ** testing a single JDK/OS permanently and push to weekly/nightly others on
> jenkins?
> ** -Reduce test matrix on maintenance stream?- DONE
> * Optimize usage of available resources
> ** see if we can build on smaller instances of Jenkins, for Linux it will
> allow to move up from 4 to ~20 instances available.
> ** use several threads (but requires a far better stability and robustness
> of tests and ensure no port conflict)
> [https://issues.apache.org/jira/browse/CAMEL-24318]
> *** to check also if we can do the pure compile in // with all threads and
> then launch tests as of now (1 threads used although 22 are available on big
> machines)
> **** on GitHub there are already 2 phases but it is using mvnd; [it is
> taking around 18
> minutes|https://github.com/apache/camel/actions/runs/30412528538/job/90451766586#step:9:1]
> **** on jenkins it means to split in different phase
> **** to test for sonar scanner ([currently 1 thread is
> used|https://develocity.apache.org/s/fjldcrmqc7bds] and it is taking between
> around 30 minutes)
> *** on Jenkins, I think we could attempt to provide a specific job for this
> experiment with specific develocity tag so that can filter out/in the results
> easily
> ** also seems 3.5G of Memory is used for the build, maybe we could use more
> when in //
> *** 32G in total on large jenkins machine
> *** but [develocity seems to report only 1.7
> max|https://develocity.apache.org/s/kuigqxgdpicsy/performance/build] is used
> currently
> *** maybe an indication that we could now run on smaller Jenkins node as
> this Memory constraint seems to date from 2018 and it has not changed, good
> chance that jenkins nodes size has been improved since that time
> * several s390x and ppc64le in //, currently [3
> ppc64le|https://jenkins-ccos.apache.org/view/Shared%20-%20ppc64le%20nodes/]
> and [6
> s390x|https://jenkins-ccos.apache.org/view/Shared%20-%20s390x%20nodes/] nodes
> are referenced as shared node but a single one is available at a time
> ** started dicussion here
> [https://the-asf.slack.com/archives/CF6PY5M4N/p1787908008691489]
> ** Bug opened in Jenkins [https://github.com/jenkinsci/jenkins/issues/27318]
> * improve performance of the product/test code itself maybe the trickier one
> but also the most beneficial for our end users
> * use of Develocity build cache (not a big fan of not rebuilding but could
> be something to reuse in some configurations)
> * use of Develocity test selection
> * improve scalpel to reduce even more the scope of PR tests
> * [DONE] improve [fetch based branch for scalpel change detection on GitHub
> which is taking almost 5
> minutes|https://github.com/apache/camel/actions/runs/30429114691/job/90501998254#step:3:1]
> * gating PR builds?
> ** have a specific account for AI agents which does not trigger build
> automatically
> ** only a subset triggered automatically then full test suite launched on
> demand for all? for specific accounts?
> * see if budget to provide more github runners and/or jenkins nodes
> * check for performance bottleneck in the build and test process (develocity
> build scan performance can give some ideas)
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
