[
https://issues.apache.org/jira/browse/FLINK-34989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17833154#comment-17833154
]
Matthias Pohl edited comment on FLINK-34989 at 4/2/24 12:18 PM:
----------------------------------------------------------------
This Jira issue is about adding job concurrency support. Ideally, we should
make it configurable in an easy way and set it to a concurrency level >20 as
requested by Apache Infra. This affects the nightly builds which run per branch
with 5 different test profiles and each test profile having 11 runners (10
stages + a short-running license check) being occupied in parallel.
Generally, we should make CI be more selective anyway. Apache Infra constantly
criticizes projects for running heavy-load CI on changes like simple doc
changes (see [here|https://infra.apache.org/github-actions-secrets.html]).
was (Author: mapohl):
This Jira issue is about adding job concurrency support. Ideally, we should
make it configurable in an easy way and set it to a concurrency level >20 as
requested by Apache Infra. This affects the nightly builds which run per branch
with 5 different test profiles and each test profile having 11 runners (10
stages + a short-running license check) being occupied in parallel.
Generally, we should make CI be more selective anyway. Apache Infra constantly
criticizes projects to run heavy-load CI for things like simple doc changes.
> Apache Infra requests to reduce the runner usage for a project
> --------------------------------------------------------------
>
> Key: FLINK-34989
> URL: https://issues.apache.org/jira/browse/FLINK-34989
> Project: Flink
> Issue Type: Sub-task
> Components: Build System / CI
> Affects Versions: 1.19.0, 1.18.1, 1.20.0
> Reporter: Matthias Pohl
> Priority: Major
> Labels: pull-request-available
>
> The GitHub Actions CI utilizes runners that are hosted by Apache Infra right
> now. These runners are limited. The runner usage can be monitored via the
> following links:
> * [Flink-specific
> report|https://infra-reports.apache.org/#ghactions&project=flink&hours=168]
> (needs ASF committer rights) This project-specific report can only be
> modified through the HTTP GET parameters of the URL.
> * [Global report|https://infra-reports.apache.org/#ghactions] (needs ASF
> membership)
> There was a policy change announced recently:
> {quote}
> Policy change on use of GitHub Actions
> Due to misconfigurations in their builds, some projects have been using
> unsupportable numbers of GitHub Actions. As part of fixing this situation,
> Infra has added a 'resource use' section to the policy on GitHub Actions.
> This section of the policy will come into effect on April 20, 2024:
> All workflows MUST have a job concurrency level less than or equal to 20.
> This means a workflow cannot have more than 20 jobs running at the same time
> across all matrices.
> All workflows SHOULD have a job concurrency level less than or equal to 15.
> Just because 20 is the max, doesn't mean you should strive for 20.
> The average number of minutes a project uses per calendar week MUST NOT
> exceed the equivalent of 25 full-time runners (250,000 minutes, or 4,200
> hours).
> The average number of minutes a project uses in any consecutive five-day
> period MUST NOT exceed the equivalent of 30 full-time runners (216,000
> minutes, or 3,600 hours).
> Projects whose builds consistently cross the maximum use limits will lose
> their access to GitHub Actions until they fix their build configurations.
> The full policy is at
> https://infra.apache.org/github-actions-policy.html.
> {quote}
> Currently (last week of March 2024) Flink was ranked at #19 of projects that
> used the Apache Infra runner resources the most which doesn't seem too bad.
> This contained not only Apache Flink but also the Kubernetes operator,
> connectors and other resources. According to [this
> source|https://infra.apache.org/github-actions-secrets.html] Apache Infra
> manages 180 runners right now.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
