[
https://issues.apache.org/jira/browse/CASSANDRA-21640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18111050#comment-18111050
]
Michael Semb Wever commented on CASSANDRA-21640:
------------------------------------------------
h4. Performance Improvements
*With the patch*
[cassandra-eks-k8s #28|https://astro-cass.ci/job/cassandra-eks-k8s/28/] ran the
full pre-commit profile, 116,633 tests, alone on the cluster: *62 minutes* end
to end. 9.1 min for jar, 45.3 min for tests, and 6.7 min for report.
Previous job's pre-commit runs:
||build||total||Tests||Summary||
|#24|56 min|41.7 min|6.9 min|
|#27|62 min|47.1 min|6.5 min|
|#28|62 min|45.3 min|6.7 min|
*Compared to pre-ci.cassandra.apache.org.*
Restricted to the pre-commit profile, on {{{}cassandra{}}}, {{cassandra-5.0}}
and {{cassandra-6.0}} between 2026-08-21 and 2026-09-03.
||build||total||Tests||Summary||share of build||
|cassandra-5.0 #192|137.2 min|58.5 min|*67.6 min*|49%|
|cassandra-6.0 #127|140.4 min|92.9 min|*32.2 min*|23%|
|cassandra-5.0 #193|133.2 min|94.6 min|*27.8 min*|21%|
|cassandra-6.0 #128|162.5 min|129.3 min|15.3 min|9%|
|cassandra-6.0 #123|81.9 min|57.9 min|10.5 min|13%|
|cassandra #694|82.7 min|62.1 min|4.9 min|6%|
|cassandra-5.0 #194|128.2 min|113.7 min|3.7 min|3%|
The main improvements here are avoiding wait time on the summary stage finding
an agent, and changing splits for burn-test form 4 to 5.
> CI: generateTestReports queues behind the test fan-out, agent pods churn
> instead of queueing, and reports on split balance
> --------------------------------------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-21640
> URL: https://issues.apache.org/jira/browse/CASSANDRA-21640
> Project: Apache Cassandra
> Issue Type: Improvement
> Components: CI
> Reporter: Michael Semb Wever
> Assignee: Michael Semb Wever
> Priority: Normal
> Fix For: 5.0.x, 6.0.x, 7.x
>
>
> Three faults in the Jenkins pipeline, each measured on a Kubernetes clone of
> ci-cassandra.apache.org. CI only; no production code changes.
> h3. 1. generateTestReports queues behind the whole test fan-out
> It runs on {{{}cassandra-medium{}}}, which is also the label of ~460 test
> splits, and it is the build's last task, so it begins only once the fan-out
> has drained: 109 minutes of a 163 minute build, for ~3 minutes of work.
> Within it, each {{xargs -P}} and {{xz -T}} sizes itself from {{{}nproc{}}},
> which answers for the node and not for the container's cgroup quota: measured
> at 8 against a 2 cpu limit, so every fan-out ran eight ways under a quota of
> two. The per-target {{ant junitreport}} loop is also serial.
> h3. 2. Agent pods churn instead of queueing
> Every podTemplate gives a pod 30 seconds to become ready
> ({{{}slaveConnectTimeout{}}}), and {{agent.waitForPodSec}} is 180. Only a
> warm node meets 30 seconds; a cold one is an instance launch, a boot, a
> kubelet join and two image pulls. Too short a deadline does not queue, it
> churns: the controller deletes the pod, the build asks again, and the nodes
> are still not there. Observed in one 24 minute window: 1,110 pods created and
> deleted, 0 agents connected, and 483 {{KubernetesProvisioningLimits ... went
> below zero}} warnings, the plugin losing count of its own caps. The cloud
> meanwhile provisions nodes for pods that no longer exist.
> Separately, the caps do not match measured demand. Over a pre-commit tests
> stage, medium held all 100 of its agents with 202 tasks queued behind them
> and large held all 160 with 467 queued, while small used 19 pods in the whole
> build and never held more than 2 at once.
> h3. 3. No build reports which cell came near its timeout
> A cell that reaches {{timeout(time: command.timeout_hours)}} is retried once
> and then fails the build with {{{}Retryable interruption: Timeout has been
> exceeded{}}}, naming neither the cell nor how close the others came.
> {{_split_tests}} deals an alphabetically sorted class list round-robin with
> {{{}split -n r/K/N{}}}, which balances the count of classes and knows nothing
> of their duration, so long classes can collide. On builds cassandra-eks-k8s
> #14 to #19, {{test-burn}} split 3 of 4 was the build's worst cell every time,
> 47.1 to 58.2 minutes against a 60 minute deadline and 2.0 to 2.6 times its
> target's median.
> h3. Patch
> https://github.com/apache/cassandra/compare/trunk...thelastpickle:cassandra:mck/ci-perf/trunk
> * {{generateTestReports}} takes a new {{cassandra-report}} label on a node
> pool of its own, added as the {{agent-dind-report}} podTemplate. A site with
> no spare nodes can put the label on its medium agents.
> * The report merge runs one {{ant junitreport}} per target, three at a time
> under an explicit {{{}-Xmx{}}}, with dind's memory limit raised to match.
> * A {{cpus()}} shell function reads the cgroup v2 quota from
> {{/proc/self/cgroup}} and {{{}cpu.max{}}}, falling back to {{nproc}} where
> there is no limit or on cgroup v1. Every {{xargs -P}} and {{xz -T}} uses it.
> * {{slaveConnectTimeout}} 600 and {{waitForPodSec}} 900, the two deadlines
> on one provisioning attempt, so both had to move. {{idleMinutes}} 5, which
> returns a node inside the autoscaler's 10 minute scale-down window.
> * {{instanceCap}} to measured demand cut to the account's quotas: small 20,
> report 4, medium 150, large 306. {{containerCap}} is their sum, so no pool's
> share is decided by which holds its agents longest, and
> {{maxRequestsPerHostStr}} scales with it.
> * New {{{}.build/ci/cell_balance.py{}}}: per target, how evenly the splits
> divided, what the worst cell ran, and how much of a cell was setup rather
> than tests. Printed to the console and appended to {{{}ci_summary.html{}}}.
> Guarded, so it never fails a build.
> * {{test-burn}} takes 5 splits. 5 divides none of the gaps between its three
> long classes, where 4 divides one of them; its worst chunk is 24.1 minutes
> against 46.7.
> * Splits stay round-robin, deliberately: which split holds a class stays
> predictable from the sorted list alone, so the lever is the split count.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
