[
https://issues.apache.org/jira/browse/CAMEL-24290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18143592#comment-18143592
]
Aurélien Pupier commented on CAMEL-24290:
-----------------------------------------
_Analysis generated by Bob (IBM Bob, AI assistant) on behalf of apupier.
Figures are derived from the Develocity build scan timeline data and should be
verified before acting on them._
h3. Critical path analysis after the camel-http speed-up (CAMEL-26819)
Scans compared:
* eb33cs73xhkq4: https://develocity.apache.org/s/eb33cs73xhkq4/timeline
* vq6u4jcg4egze: https://develocity.apache.org/s/vq6u4jcg4egze/timeline
* elj7p4rvx65gi (after the camel-http change):
https://develocity.apache.org/s/elj7p4rvx65gi/timeline
h4. Method
* Per-module span = first goal start to last goal end, taken from
{{/scan-data/maven/<scan>/timeline}}.
* Dependency edges inferred from {{pom.xml}} files (parent, dependencies and
plugins, with parent inheritance).
* Critical path = start at the last-finishing module and repeatedly step back
to the dependency that finished last before the module started. The path has 51
modules, and the gap between consecutive steps is at most 3.6 s.
* CPU = mean of the machine-wide {{allProcessesCpu}} metric (about 1 s
sampling) over each module's span. "% time <=60% CPU" is the share of the
module's time at or below 60% load. Idle-seconds = duration x (100 - mean CPU%).
h4. Result for camel-http
||Scan||Build||HTTP module span||surefire default-test||surefire
isolated-test||CPU %||
|eb33cs73xhkq4|2303.4 s|314.4 s|279 s|25 s|98.4|
|vq6u4jcg4egze|2278.6 s|315.5 s|287 s|17 s|98.3|
|*elj7p4rvx65gi*|*2227.6 s*|*90.8 s*|*58 s*|22 s|98.7|
* camel-http is no longer on the critical path.
* The build total improved by about 51 s versus the previous scan, not by the
full ~225 s saved in HTTP. {{camel-netty}} starts at the same time as HTTP (924
s) and takes 277 s ({{surefire:test}} 271 s), so it now blocks
{{camel-netty-http}}.
h4. Critical path modules of at least 10 s (scan elj7p4rvx65gi, path order)
||Module||Start (s)||Duration||CPU %||% time <=60% CPU||Idle-s||Longest goal||
|Util|35|10.7 s|25|90|8|surefire:test 6 s|
|Support|53|13.5 s|18|100|11|compiler:compile 3 s|
|Core Model|81|10.2 s|13|100|9|compiler:compile 2 s|
|XML IO|92|23.0 s|60|50|9|jaxb2:schemagen 6 s|
|*Core*|120|*250.9 s*|16|93|*210*|surefire:test 213 s|
|Management|371|82.4 s|21|100|65|surefire:test 74 s|
|Test :: JUnit6|454|17.9 s|50|61|9|surefire:test 16 s|
|Zip Deflater to Simple JMS (7 modules)|472 to 822|12 to 80 s
each|91-98|0-8|0-2|surefire:test 10-77 s|
|Spring XML|823|65.5 s|75|23|16|surefire:test 41 s|
|Test :: Spring :: JUnit6|888|35.2 s|65|40|12|surefire:test 34 s|
|*Netty*|924|*277.3 s*|99|0|4|surefire:test 271 s|
|Jetty|1201|70.9 s|96|0|3|surefire:test 62 s|
|*Netty HTTP*|1272|*278.1 s*|92|3|23|surefire:test 270 s|
|REST OpenApi|1550|16.6 s|49|88|9|surefire:test 15 s|
|Platform HTTP :: Vert.x|1567|21.9 s|12|100|19|surefire:test 20 s|
|Platform HTTP :: Main|1589|10.5 s|67|30|4|surefire:test 9 s|
|OAuth|1599|34.1 s|42|76|20|surefire:test 32 s|
|*AI :: MCP Server*|1633|*101.2 s*|27|97|*74*|failsafe:integration-test 74 s|
|Catalog :: Camel Catalog|1737|26.8 s|17|100|22|surefire:test 21 s|
|YAML DSL|1771|20.4 s|45|76|11|surefire:test 15 s|
|JBang :: Run|1802|13.0 s|57|46|6|surefire:test 9 s|
|*JBang :: Core*|1815|*271.8 s*|17|98|*225*|surefire:test serial-test 174 s|
|JBang :: Plugin :: TUI|2087|67.2 s|31|73|46|surefire:test 41 s|
|Launcher|2154|68.8 s|22|96|54|surefire:test 55 s|
h4. Long modules that also have CPU available (ranked by idle-seconds)
# *JBang :: Core*: 272 s at 17% CPU, 225 idle-s. The {{serial-test}} execution
is 174 s of it.
# *Core*: 251 s at 16% CPU, 210 idle-s.
# *AI :: MCP Server*: 101 s at 27% CPU, 74 idle-s.
# *Management*: 82 s at 21% CPU, 65 idle-s.
# *Launcher*: 69 s and *JBang :: Plugin :: TUI*: 67 s, with 54 and 46 idle-s.
h4. Long modules where the machine is saturated
* *Netty* (277 s), *Netty HTTP* (278 s) and Jetty (71 s), plus the stretch from
Zip Deflater to Simple JMS (472 to 822 s, about 330 s at 91-98% CPU). These are
CPU-bound, so running their tests in parallel is unlikely to help. Only lighter
tests would.
h4. Observations on other modules (across the three scans)
||Module||eb33cs73xhkq4||vq6u4jcg4egze||elj7p4rvx65gi||
|Netty HTTP span|281.7 s|309.5 s|278.1 s|
|Jetty span|56.5 s|54.4 s|70.9 s|
|Core span|265.9 s|243.3 s|250.9 s|
|JBang :: Core span|281.1 s|304.1 s|271.8 s|
|Spring XML span|116.6 s|67.8 s|65.5 s|
|AI :: MCP Server span|59.6 s|63.6 s|101.2 s|
* AI :: MCP Server: {{failsafe:integration-test}} went from 34 s to 74 s
between the last two scans. It has not been checked whether this is a real
regression or noise (for example container startup).
* Jetty went from 54 s to 71 s. This may be contention from running alongside
the now-faster HTTP module.
h4. Caveats
* Dependency edges are inferred from pom files, not reported by Develocity. The
spine Core -> Netty -> Netty HTTP -> JBang :: Core is well supported. The
stretch between about 454 s and 822 s depends on Maven {{-T1C}} scheduling and
is not a fixed dependency chain, so it differs between scans.
* CPU is machine-wide (all processes on the agent), not per module, and sampled
at about 1 s.
* The data does not show why a module leaves CPU idle (serial tests, sleeps or
timeouts). That needs a look at the surefire configuration and per-test
durations.
> Improve Continuous Integration to support larger workload
> ---------------------------------------------------------
>
> Key: CAMEL-24290
> URL: https://issues.apache.org/jira/browse/CAMEL-24290
> Project: Camel
> Issue Type: Task
> Components: build system
> Affects Versions: 4.21.0
> Reporter: Aurélien Pupier
> Priority: Major
>
> Our 2 CI systems (Jenkins and GitHub actions) are already under heavy loads
> and the planned future will surely increase the load and the need for a more
> stable CI.
> Currently Jenkins is taking between 4 and 5 hours when things are going well.
> Can go up to 20 hours when all streams in //
> On GitHub, it happens regularly that the jobs are queued for several hours.
> We have few full green build per week on Jenkins. (We are at one or two per
> year last year, so already a lot better but not enough to really sustain new
> workloads efficiently).
> Having a CI failing only for real issues will also help all committers to
> react faster
>
> There are several factors for increased, and potential new increase:
> * Increased number PRs with more review, so more cycle
> * Increased number of backports
> * Increased cadence of dependencies release
> * Longer LTS ? , more streams to maintain?
> * Shared GitHub runners with all Apache projects (and several of them are
> facing the same workload increase)
> Several areas identified which worth at least some investigations:
> * stability of Workflows run: it gives more confidence to the build, avoids
> the need to rerun to check that it is "just" flaky or an infra hiccups
> ** develocity help tracking flaky tests (especially flaky tests causing
> failure)
> ** -one last test is failing very often: SjmsConnectionRecoveryTest-
> -[https://issues.apache.org/jira/browse/CAMEL-24284-] DONE no failure since
> 1st September
> ** several failing less often (every months)
> *** useful Develocity dashboard
> [https://develocity.apache.org/scans/tests?search.query=project:camel%20tag:main%20-maven.environment.jvmVersion:*26*&search.relativeStartTime=P28D]
> **** beware of tests which failed for good reasons (it caught regressions)
> **** beware of all the tests which were already fixed (there was a huge
> session of fix, so maybe need to wait a month to have a clearer view and/or
> play with the the timeframe)
> **** I filtered on main and excluded JDK 26 for now. Because i have JDK 26
> has tests which are failing for an identified bug and a lot of stability
> improvements were done on main only
> **** based on previous warning, the current curated list is:
> ***** MasterEndpointIT
> [https://develocity.apache.org/scans/tests?search.query=project:camel%20tag:main%20-maven.environment.jvmVersion:*26*&search.timeZoneId=Europe%2FParis&tests.container=org.apache.camel.component.zookeepermaster.MasterEndpointIT&tests.test=testEndpoint]
> ***** MongoDB:
> [MongoDbBulkWriteOperationsIT|https://develocity.apache.org/scans/tests?search.query=project:camel%20tag:main%20-maven.environment.jvmVersion:*26*&search.timeZoneId=Europe%2FParis&tests.container=org.apache.camel.component.mongodb.integration.MongoDbBulkWriteOperationIT],
>
> [MongoDbCredentialsFromUriConnectionIT|https://develocity.apache.org/scans/tests?search.query=project:camel%20tag:main%20-maven.environment.jvmVersion:*26*&search.timeZoneId=Europe%2FParis&tests.container=org.apache.camel.component.mongodb.integration.MongoDbCredentialsFromUriConnectionIT]
> and
> [MongoDbFindOneAndReplaceOperationIT|https://develocity.apache.org/scans/tests?search.query=project:camel%20tag:main%20-maven.environment.jvmVersion:*26*&search.timeZoneId=Europe%2FParis&tests.container=org.apache.camel.component.mongodb.integration.MongoDbFindOneAndReplaceOperationIT]
> *****
> [SpringFileAntPathMatcherRemoteFileFilterTest|https://develocity.apache.org/scans/tests?search.query=project:camel%20tag:main%20-maven.environment.jvmVersion:*26*&search.timeZoneId=Europe%2FParis&tests.container=org.apache.camel.itest.ftp.SpringFileAntPathMatcherRemoteFileFilterTest]
> CAMEL-24618
> *****
> [CassandraIdempotentRespotoryIT|https://develocity.apache.org/scans/tests?search.query=project:camel%20tag:main%20-maven.environment.jvmVersion:*26*&search.timeZoneId=Europe%2FParis&tests.container=org.apache.camel.processor.idempotent.cassandra.CassandraIdempotentRepositoryIT]
> ***** and certainly a few others currently buried in the list of already
> fixed ones
> ** and also several flaky during the run but passing in less than 3
> attempts, slowing down the build
> ***
> [https://develocity.apache.org/scans/tests?search.query=project:camel%20tag:main&search.relativeStartTime=P28D&search.timeZoneId=Europe%2FParis&tests.sortField=FLAKY]
> ***
> ** -one Jenkins workspace leak to fix due to some jbang tests- CAMEL-23795
> DONE
> * modification of test matrixes
> ** testing a single JDK/OS permanently and push to weekly/nightly others on
> jenkins?
> ** -Reduce test matrix on maintenance stream?- DONE
> * Optimize usage of available resources
> ** see if we can build on smaller instances of Jenkins, for Linux it will
> allow to move up from 4 to ~20 instances available.
> ** use several threads (but requires a far better stability and robustness
> of tests and ensure no port conflict)
> [https://issues.apache.org/jira/browse/CAMEL-24318]
> *** to check also if we can do the pure compile in // with all threads and
> then launch tests as of now (1 threads used although 22 are available on big
> machines)
> **** on GitHub there are already 2 phases but it is using mvnd; [it is
> taking around 18
> minutes|https://github.com/apache/camel/actions/runs/30412528538/job/90451766586#step:9:1]
> **** on jenkins it means to split in different phase
> **** to test for sonar scanner ([currently 1 thread is
> used|https://develocity.apache.org/s/fjldcrmqc7bds] and it is taking between
> around 30 minutes)
> *** on Jenkins, I think we could attempt to provide a specific job for this
> experiment with specific develocity tag so that can filter out/in the results
> easily
> ** also seems 3.5G of Memory is used for the build, maybe we could use more
> when in //
> *** 32G in total on large jenkins machine
> *** but [develocity seems to report only 1.7
> max|https://develocity.apache.org/s/kuigqxgdpicsy/performance/build] is used
> currently
> *** maybe an indication that we could now run on smaller Jenkins node as
> this Memory constraint seems to date from 2018 and it has not changed, good
> chance that jenkins nodes size has been improved since that time
> * several s390x and ppc64le in //, currently [3
> ppc64le|https://jenkins-ccos.apache.org/view/Shared%20-%20ppc64le%20nodes/]
> and [6
> s390x|https://jenkins-ccos.apache.org/view/Shared%20-%20s390x%20nodes/] nodes
> are referenced as shared node but a single one is available at a time
> ** started dicussion here
> [https://the-asf.slack.com/archives/CF6PY5M4N/p1787908008691489]
> ** Bug opened in Jenkins [https://github.com/jenkinsci/jenkins/issues/27318]
> * improve performance of the product/test code itself maybe the trickier one
> but also the most beneficial for our end users
> * use of Develocity build cache (not a big fan of not rebuilding but could
> be something to reuse in some configurations)
> * use of Develocity test selection
> * improve scalpel to reduce even more the scope of PR tests
> * [DONE] improve [fetch based branch for scalpel change detection on GitHub
> which is taking almost 5
> minutes|https://github.com/apache/camel/actions/runs/30429114691/job/90501998254#step:3:1]
> * gating PR builds?
> ** have a specific account for AI agents which does not trigger build
> automatically
> ** only a subset triggered automatically then full test suite launched on
> demand for all? for specific accounts?
> * see if budget to provide more github runners and/or jenkins nodes
> * check for performance bottleneck in the build and test process (develocity
> build scan performance can give some ideas)
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
