[
https://issues.apache.org/jira/browse/CASSANDRA-21640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18111932#comment-18111932
]
Dmitry Konstantinov commented on CASSANDRA-21640:
-------------------------------------------------
+1, thank for making CI better and better
> CI: generateTestReports queues behind the test fan-out, agent pods churn
> instead of queueing, and reports on split balance
> --------------------------------------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-21640
> URL: https://issues.apache.org/jira/browse/CASSANDRA-21640
> Project: Apache Cassandra
> Issue Type: Improvement
> Components: CI
> Reporter: Michael Semb Wever
> Assignee: Michael Semb Wever
> Priority: Normal
> Fix For: 5.0.x, 6.0.x, 7.x
>
> Attachments: ci_summary-10.html, ci_summary-8.html,
> results_details.tar-10.xz, results_details.tar-8.xz
>
>
> Three faults in the Jenkins pipeline, each measured on a Kubernetes clone of
> ci-cassandra.apache.org. CI only; no production code changes.
> h3. 1. generateTestReports queues behind the whole test fan-out
> It runs on {{{}cassandra-medium{}}}, which is also the label of ~460 test
> splits, and it is the build's last task, so it begins only once the fan-out
> has drained: 109 minutes of a 163 minute build, for ~3 minutes of work.
> Within it, each {{xargs -P}} and {{xz -T}} sizes itself from {{{}nproc{}}},
> which answers for the node and not for the container's cgroup quota: measured
> at 8 against a 2 cpu limit, so every fan-out ran eight ways under a quota of
> two. The per-target {{ant junitreport}} loop is also serial.
> h3. 2. Agent pods churn instead of queueing
> Every podTemplate gives a pod 30 seconds to become ready
> ({{{}slaveConnectTimeout{}}}), and {{agent.waitForPodSec}} is 180. Only a
> warm node meets 30 seconds; a cold one is an instance launch, a boot, a
> kubelet join and two image pulls. Too short a deadline does not queue, it
> churns: the controller deletes the pod, the build asks again, and the nodes
> are still not there. Observed in one 24 minute window: 1,110 pods created and
> deleted, 0 agents connected, and 483 {{KubernetesProvisioningLimits ... went
> below zero}} warnings, the plugin losing count of its own caps. The cloud
> meanwhile provisions nodes for pods that no longer exist.
> Separately, the caps do not match measured demand. Over a pre-commit tests
> stage, medium held all 100 of its agents with 202 tasks queued behind them
> and large held all 160 with 467 queued, while small used 19 pods in the whole
> build and never held more than 2 at once.
> h3. 3. No build reports which cell came near its timeout
> A cell that reaches {{timeout(time: command.timeout_hours)}} is retried once
> and then fails the build with {{{}Retryable interruption: Timeout has been
> exceeded{}}}, naming neither the cell nor how close the others came.
> {{_split_tests}} deals an alphabetically sorted class list round-robin with
> {{{}split -n r/K/N{}}}, which balances the count of classes and knows nothing
> of their duration, so long classes can collide. On builds cassandra-eks-k8s
> #14 to #19, {{test-burn}} split 3 of 4 was the build's worst cell every time,
> 47.1 to 58.2 minutes against a 60 minute deadline and 2.0 to 2.6 times its
> target's median.
> h3. Patch
> https://github.com/apache/cassandra/compare/trunk...thelastpickle:cassandra:mck/ci-perf/trunk
> * {{generateTestReports}} takes a new {{cassandra-report}} label on a node
> pool of its own, added as the {{agent-dind-report}} podTemplate. A site with
> no spare nodes can put the label on its medium agents.
> * The report merge runs one {{ant junitreport}} per target, three at a time
> under an explicit {{{}-Xmx{}}}, with dind's memory limit raised to match.
> * A {{cpus()}} shell function reads the cgroup v2 quota from
> {{/proc/self/cgroup}} and {{{}cpu.max{}}}, falling back to {{nproc}} where
> there is no limit or on cgroup v1. Every {{xargs -P}} and {{xz -T}} uses it.
> * {{slaveConnectTimeout}} 600 and {{waitForPodSec}} 900, the two deadlines
> on one provisioning attempt, so both had to move. {{idleMinutes}} 5, which
> returns a node inside the autoscaler's 10 minute scale-down window.
> * {{instanceCap}} to measured demand cut to the account's quotas: small 20,
> report 4, medium 150, large 306. {{containerCap}} is their sum, so no pool's
> share is decided by which holds its agents longest, and
> {{maxRequestsPerHostStr}} scales with it.
> * New {{{}.build/ci/cell_balance.py{}}}: per target, how evenly the splits
> divided, what the worst cell ran, and how much of a cell was setup rather
> than tests. Printed to the console and appended to {{{}ci_summary.html{}}}.
> Guarded, so it never fails a build.
> * {{test-burn}} takes 5 splits. 5 divides none of the gaps between its three
> long classes, where 4 divides one of them; its worst chunk is 24.1 minutes
> against 46.7.
> * Splits stay round-robin, deliberately: which split holds a class stays
> predictable from the sorted list alone, so the lever is the split count.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
