[
https://issues.apache.org/jira/browse/SPARK-61159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ruifeng Zheng updated SPARK-61159:
----------------------------------
Description:
ASF Infrastructure has asked Spark to reduce GitHub-hosted Actions runner
consumption and concurrent job pressure.
Reduce and stagger scheduled workloads without making any build variant more
frequent. Move the 52 two-day variants to three days, preserve all original
four- and six-day cadences, and keep the base-image cache daily at 04:17 UTC.
Preserve daily Pages at 12:30 UTC, daily stale cleanup at 00:30 UTC, the
complete original branch-3.5 scheduler, and the build-status updater.
Reserve UTC hours 00, 04, 08, 11, 12, and 20 for workloads outside the
three-day cadence. Use 04:17, 08:17, and 20:17 for four- and six-day variants.
The 52 three-day workloads use separate slots in the remaining 18 hours. Minute
17 avoids GitHub's busy start-of-hour scheduling period; it is an arbitrary
off-hour choice.
Keep day steps at least as long as the original interval. Four-day and six-day
variants use date offsets within their reserved hours. Occasional overlap is
allowed when these cycles intersect. Day-of-month offsets restart monthly, so
the cadences are approximate calendar intervals rather than strict rolling
intervals.
The affected workloads are not ranked by importance. Cadence follows the
existing interval with a three-day minimum for rescheduled builds, while
established maintenance exceptions are preserved. An individual interval can be
shortened later if faster feedback is needed.
Update the matching branch-dispatch conditions, preserving manual dispatch,
target branches, other event triggers, and skip-if-unchanged checks. Document
the reserved hours, cadence policy, minute choice, and allowed overlap in
.github/workflows/README.md.
Estimated savings are about 30% of aggregate scheduled runner-minutes and 17%
of total CI runner-minutes at the observed usage mix, assuming similar
durations and branch activity. Individual jobs do not execute faster because of
this change.
PR: https://github.com/apache/spark/pull/60216
Validation: Actionlint, before/after YAML comparisons, non-decreasing interval
checks, reserved-hour checks, and 2026-2032 calendar enumeration. There are 52
three-day, 11 four-day, four six-day, and three daily target workloads. At most
two distinct triggers coincide, at reserved 04:17 daily and reserved 20:17 on
dates 9 and 21.
was:
ASF Infrastructure has asked Spark to reduce GitHub-hosted Actions runner
consumption and concurrent job pressure.
Reduce and stagger scheduled workloads without making any build variant more
frequent. Move the 52 two-day variants to three days, preserve all original
four- and six-day cadences, and move the daily base-image cache rebuild to
three days. Preserve daily Pages at 12:30 UTC, daily stale cleanup at 00:30
UTC, the complete original branch-3.5 scheduler, and the build-status updater.
Reserve UTC hours 00, 04, 08, 11, 12, and 20 for workloads outside the
three-day cadence. Use 04:17, 08:17, and 20:17 for four- and six-day variants.
The 53 three-day workloads use separate slots in the remaining 18 hours. Minute
17 avoids GitHub's busy start-of-hour scheduling period; it is an arbitrary
off-hour choice.
Keep day steps at least as long as the original interval. Four-day and six-day
variants use date offsets within their reserved hours. Occasional overlap is
allowed when these cycles intersect. Day-of-month offsets restart monthly, so
the cadences are approximate calendar intervals rather than strict rolling
intervals.
The affected workloads are not ranked by importance. Cadence follows the
existing interval with a three-day minimum for rescheduled builds, while
established maintenance exceptions are preserved. An individual interval can be
shortened later if faster feedback is needed.
Update the matching branch-dispatch conditions, preserving manual dispatch,
target branches, other event triggers, and skip-if-unchanged checks. Document
the reserved hours, cadence policy, minute choice, and allowed overlap in
.github/workflows/README.md.
Estimated savings are about 30% of aggregate scheduled runner-minutes and 17%
of total CI runner-minutes at the observed usage mix, assuming similar
durations and branch activity. Individual jobs do not execute faster because of
this change.
PR: https://github.com/apache/spark/pull/60216
Validation: Actionlint, before/after YAML comparisons, non-decreasing interval
checks, reserved-hour checks, and 2026-2032 calendar enumeration. There are 53
three-day, 11 four-day, four six-day, and two daily target workloads. At most
two distinct triggers coincide, only at reserved 20:17 on dates 9 and 21.
> Reduce and stagger scheduled workflow runs
> ------------------------------------------
>
> Key: SPARK-61159
> URL: https://issues.apache.org/jira/browse/SPARK-61159
> Project: Spark
> Issue Type: Improvement
> Components: Project Infra
> Affects Versions: 5.0.0
> Reporter: Ruifeng Zheng
> Priority: Major
>
> ASF Infrastructure has asked Spark to reduce GitHub-hosted Actions runner
> consumption and concurrent job pressure.
> Reduce and stagger scheduled workloads without making any build variant more
> frequent. Move the 52 two-day variants to three days, preserve all original
> four- and six-day cadences, and keep the base-image cache daily at 04:17 UTC.
> Preserve daily Pages at 12:30 UTC, daily stale cleanup at 00:30 UTC, the
> complete original branch-3.5 scheduler, and the build-status updater.
> Reserve UTC hours 00, 04, 08, 11, 12, and 20 for workloads outside the
> three-day cadence. Use 04:17, 08:17, and 20:17 for four- and six-day
> variants. The 52 three-day workloads use separate slots in the remaining 18
> hours. Minute 17 avoids GitHub's busy start-of-hour scheduling period; it is
> an arbitrary off-hour choice.
> Keep day steps at least as long as the original interval. Four-day and
> six-day variants use date offsets within their reserved hours. Occasional
> overlap is allowed when these cycles intersect. Day-of-month offsets restart
> monthly, so the cadences are approximate calendar intervals rather than
> strict rolling intervals.
> The affected workloads are not ranked by importance. Cadence follows the
> existing interval with a three-day minimum for rescheduled builds, while
> established maintenance exceptions are preserved. An individual interval can
> be shortened later if faster feedback is needed.
> Update the matching branch-dispatch conditions, preserving manual dispatch,
> target branches, other event triggers, and skip-if-unchanged checks. Document
> the reserved hours, cadence policy, minute choice, and allowed overlap in
> .github/workflows/README.md.
> Estimated savings are about 30% of aggregate scheduled runner-minutes and 17%
> of total CI runner-minutes at the observed usage mix, assuming similar
> durations and branch activity. Individual jobs do not execute faster because
> of this change.
> PR: https://github.com/apache/spark/pull/60216
> Validation: Actionlint, before/after YAML comparisons, non-decreasing
> interval checks, reserved-hour checks, and 2026-2032 calendar enumeration.
> There are 52 three-day, 11 four-day, four six-day, and three daily target
> workloads. At most two distinct triggers coincide, at reserved 04:17 daily
> and reserved 20:17 on dates 9 and 21.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
