[
https://issues.apache.org/jira/browse/SPARK-61159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ruifeng Zheng updated SPARK-61159:
----------------------------------
Description:
ASF Infrastructure has asked Spark to reduce GitHub-hosted Actions runner
consumption and concurrent job pressure.
Reduce and stagger scheduled workloads without making any build variant more
frequent. Move the 52 two-day variants to three days, preserve all original
four- and six-day cadences, and move the daily base-image cache rebuild to
three days. Preserve daily Pages at 12:30 UTC, daily stale cleanup at 00:30
UTC, the complete original branch-3.5 scheduler, and the build-status updater.
Reserve UTC hours 00, 04, 08, 11, 12, and 20 for workloads outside the
three-day cadence. Use 04:17, 08:17, and 20:17 for four- and six-day variants.
The 53 three-day workloads use separate slots in the remaining 18 hours. Minute
17 avoids GitHub's busy start-of-hour scheduling period; it is an arbitrary
off-hour choice.
Keep day steps at least as long as the original interval. Four-day and six-day
variants use date offsets within their reserved hours. Occasional overlap is
allowed when these cycles intersect. Day-of-month offsets restart monthly, so
the cadences are approximate calendar intervals rather than strict rolling
intervals.
The affected workloads are not ranked by importance. Cadence follows the
existing interval with a three-day minimum for rescheduled builds, while
established maintenance exceptions are preserved. An individual interval can be
shortened later if faster feedback is needed.
Update the matching branch-dispatch conditions, preserving manual dispatch,
target branches, other event triggers, and skip-if-unchanged checks. Document
the reserved hours, cadence policy, minute choice, and allowed overlap in
.github/workflows/README.md.
Estimated savings are about 30% of aggregate scheduled runner-minutes and 17%
of total CI runner-minutes at the observed usage mix, assuming similar
durations and branch activity. Individual jobs do not execute faster because of
this change.
PR: https://github.com/apache/spark/pull/60216
Validation: Actionlint, before/after YAML comparisons, non-decreasing interval
checks, reserved-hour checks, and 2026-2032 calendar enumeration. There are 53
three-day, 11 four-day, four six-day, and two daily target workloads. At most
two distinct triggers coincide, only at reserved 20:17 on dates 9 and 21.
was:
ASF Infrastructure has asked Spark to reduce GitHub-hosted Actions runner
consumption and concurrent job pressure. Many scheduled variants currently run
on the same odd dates, with several sharing an hour.
Move 66 scheduled workloads on master and maintenance branches to three
day-of-month phases, staggered by hour. Use minute 17 to avoid GitHub's busy
start-of-hour scheduling period; 17 is an arbitrary off-hour choice. The phases
reset monthly, so the interval is approximately three days rather than a strict
rolling 72 hours.
Keep Pages daily at 12:30 UTC and stale-PR cleanup daily at 00:30 UTC. Preserve
the original branch-3.5 scheduler, including its four-day cadence and shared
11:00 UTC trigger. Leave the build-status updater unchanged. Update all
matching branch-dispatch conditions and preserve manual dispatch, target
branches, and skip-if-unchanged checks.
The affected variants share a common cadence without ranking jobs by
importance. Their phase and hour assignments spread runner demand. An
individual interval can be shortened later if faster feedback is needed.
Estimated reduction in aggregate scheduled runner-minutes is 25-27%, or 14-15%
of total CI runner-minutes at the observed usage mix. These are projections
assuming similar job durations and branch activity, including the increases for
four- and six-day variants. Individual jobs do not execute faster because of
this change.
PR: https://github.com/apache/spark/pull/60216
Validation: Actionlint, before/after YAML comparisons, branch-dispatch mapping
checks, and calendar enumeration from 2026 through 2032.
Summary: Reduce and stagger scheduled workflow runs (was: Stagger
scheduled workflows on a three-day cadence)
> Reduce and stagger scheduled workflow runs
> ------------------------------------------
>
> Key: SPARK-61159
> URL: https://issues.apache.org/jira/browse/SPARK-61159
> Project: Spark
> Issue Type: Improvement
> Components: Project Infra
> Affects Versions: 5.0.0
> Reporter: Ruifeng Zheng
> Priority: Major
>
> ASF Infrastructure has asked Spark to reduce GitHub-hosted Actions runner
> consumption and concurrent job pressure.
> Reduce and stagger scheduled workloads without making any build variant more
> frequent. Move the 52 two-day variants to three days, preserve all original
> four- and six-day cadences, and move the daily base-image cache rebuild to
> three days. Preserve daily Pages at 12:30 UTC, daily stale cleanup at 00:30
> UTC, the complete original branch-3.5 scheduler, and the build-status updater.
> Reserve UTC hours 00, 04, 08, 11, 12, and 20 for workloads outside the
> three-day cadence. Use 04:17, 08:17, and 20:17 for four- and six-day
> variants. The 53 three-day workloads use separate slots in the remaining 18
> hours. Minute 17 avoids GitHub's busy start-of-hour scheduling period; it is
> an arbitrary off-hour choice.
> Keep day steps at least as long as the original interval. Four-day and
> six-day variants use date offsets within their reserved hours. Occasional
> overlap is allowed when these cycles intersect. Day-of-month offsets restart
> monthly, so the cadences are approximate calendar intervals rather than
> strict rolling intervals.
> The affected workloads are not ranked by importance. Cadence follows the
> existing interval with a three-day minimum for rescheduled builds, while
> established maintenance exceptions are preserved. An individual interval can
> be shortened later if faster feedback is needed.
> Update the matching branch-dispatch conditions, preserving manual dispatch,
> target branches, other event triggers, and skip-if-unchanged checks. Document
> the reserved hours, cadence policy, minute choice, and allowed overlap in
> .github/workflows/README.md.
> Estimated savings are about 30% of aggregate scheduled runner-minutes and 17%
> of total CI runner-minutes at the observed usage mix, assuming similar
> durations and branch activity. Individual jobs do not execute faster because
> of this change.
> PR: https://github.com/apache/spark/pull/60216
> Validation: Actionlint, before/after YAML comparisons, non-decreasing
> interval checks, reserved-hour checks, and 2026-2032 calendar enumeration.
> There are 53 three-day, 11 four-day, four six-day, and two daily target
> workloads. At most two distinct triggers coincide, only at reserved 20:17 on
> dates 9 and 21.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
