On Fri, Apr 26, 2024 at 12:02 AM Nazir Bilal Yavuz <byavu...@gmail.com> wrote:
> On Sat, 13 Apr 2024 at 05:12, Andres Freund <and...@anarazel.de> wrote:
> > I propose that we instead run the task automatically if
> > $REPO_MINGW_TRIGGER_BY_DEFAULT is set, typically in cirrus' per-repository
> > configuration.
> >
> > Unfortunately that's somewhat awkward to do in the cirrus-ci yaml
> > configuration, the set of conditional expressions supported is very
> > simplistic.
> >
> > To deal with that, I extended .cirrus.star to compute the required
> > environment
> > variable. If $REPO_MINGW_TRIGGER_BY_DEFAULT is set, CI_MINGW_TRIGGER_TYPE is
> > set to 'automatic', if not it's 'manual'.
>
> Changes look good to me. My only complaint could be using only 'true'
> for the REPO_MINGW_TRIGGER_BY_DEFAULT, not a possible list of values
> but this is not important.
Here's a generalised version of 0001. If you put this in your
repository settings on Cirrus's website:
REPO_CI_AUTOMATIC_TRIGGER_TASKS="mingw netbsd"
... then it defines:
CI_TRIGGER_TYPE_MINGW=automatic
CI_TRIGGER_TYPE_NETBSD=automatic
CI_TRIGGER_TYPE_OPENBSD=manual
The set of tasks that get this treatment is defined by this line in
.cirrus.star:
default_manual_trigger_tasks = ['mingw', 'netbsd', 'openbsd']
Then the individual tasks in .cirrus.tasks.yml use those variables:
- name: NetBSD - Meson
+ # See REPO_CI_AUTOMATIC_TRIGGER_TASKS in .cirrus.star
+ trigger_type: $CI_TRIGGER_TYPE_NETBSD
Unfortunately the funky Starlark language doesn't seem to have regular
expressions, so it's just searching for those substrings without
contemplating delimiters. That doesn't seem to be a problem at this
scale...
(I didn't look at the second patch today.)
From 74f4fb7a5af0b53eb4caf7de0ee186ffaaf6ee20 Mon Sep 17 00:00:00 2001
From: Andres Freund <and...@anarazel.de>
Date: Fri, 12 Apr 2024 15:04:32 -0700
Subject: [PATCH v3] ci: Per-repo triggers for MinGW, NetBSD, OpenBSD.
We're not ready to trigger these tasks automatically by default, so you
normally have to trigger then manually via Cirrus if you want to run
them. This avoids using too many credits by default.
With this commit, an individual repository can be configured to trigger
them automatically using an environment variable defined under
"Repository Settings", for example:
REPO_CI_AUTOMATIC_TRIGGER_TASKS="mingw netbsd openbsd"
This will enable cfbot to turn them on by default when running tests for
the Commitfest app. It is not subject to free credit limits as it runs
on credits donated by Google.
Author: Andres Freund <and...@anarazel.de>
Co-authored-by: Thomas Munro <thomas.mu...@gmail.com>
Reviewed-by: Nazir Bilal Yavuz <byavu...@gmail.com>
Discussion: https://postgr.es/m/20240413021221.hg53rvqlvldqh57i%40awork3.anarazel.de
---
.cirrus.star | 50 +++++++++++++++++++++++++++++++++++++++++----
.cirrus.tasks.yml | 15 +++++++-------
.cirrus.yml | 12 +++++++++--
src/tools/ci/README | 11 ++++++++++
4 files changed, 75 insertions(+), 13 deletions(-)
diff --git a/.cirrus.star b/.cirrus.star
index 36233872d1e..218fce887b5 100644
--- a/.cirrus.star
+++ b/.cirrus.star
@@ -7,7 +7,7 @@ https://github.com/bazelbuild/starlark/blob/master/spec.md
See also .cirrus.yml and src/tools/ci/README
"""
-load("cirrus", "env", "fs")
+load("cirrus", "env", "fs", "yaml")
def main():
@@ -18,19 +18,36 @@ def main():
1) the contents of .cirrus.yml
- 2) if defined, the contents of the file referenced by the, repository
+ 2) computed environment variables
+
+ 3) if defined, the contents of the file referenced by the, repository
level, REPO_CI_CONFIG_GIT_URL variable (see
https://cirrus-ci.org/guide/programming-tasks/#fs for the accepted
format)
- 3) .cirrus.tasks.yml
+ 4) .cirrus.tasks.yml
"""
output = ""
# 1) is evaluated implicitly
+
# Add 2)
+ additional_env = compute_environment_vars()
+ env_fmt = """
+###
+# Computed environment variables start here
+###
+{0}
+###
+# Computed environment variables end here
+###
+"""
+ output += env_fmt.format(yaml.dumps({'env': additional_env}))
+
+
+ # Add 3)
repo_config_url = env.get("REPO_CI_CONFIG_GIT_URL")
if repo_config_url != None:
print("loading additional configuration from \"{}\"".format(repo_config_url))
@@ -38,12 +55,37 @@ def main():
else:
output += "\n# REPO_CI_CONFIG_URL was not set\n"
- # Add 3)
+
+ # Add 4)
output += config_from(".cirrus.tasks.yml")
+
return output
+def compute_environment_vars():
+ cenv = {}
+
+ # The following tasks are manually triggered by default because they
+ # might use too many resources for users of free Cirrus credits, but you can
+ # trigger them automatically by naming them in an environment variable eg
+ # REPO_CI_AUTOMATIC_TRIGGER_TASKS="mingw netbsd" under "Repository
+ # Settings" on Cirrus CI's website.
+
+ default_manual_trigger_tasks = ['mingw', 'netbsd', 'openbsd']
+
+ repo_ci_automatic_trigger_tasks = env.get('REPO_CI_AUTOMATIC_TRIGGER_TASKS', '')
+ for task in default_manual_trigger_tasks:
+ name = 'CI_TRIGGER_TYPE_' + task.upper()
+ if repo_ci_automatic_trigger_tasks.find(task) != -1:
+ value = 'automatic'
+ else:
+ value = 'manual'
+ cenv[name] = value
+
+ return cenv
+
+
def config_from(config_src):
"""return contents of config file `config_src`, surrounded by markers
indicating start / end of the included file
diff --git a/.cirrus.tasks.yml b/.cirrus.tasks.yml
index c5e7b743bfb..f5ff0ccf222 100644
--- a/.cirrus.tasks.yml
+++ b/.cirrus.tasks.yml
@@ -218,7 +218,6 @@ task:
task:
depends_on: SanityCheck
- trigger_type: manual
env:
# Below are experimentally derived to be a decent choice.
@@ -236,6 +235,8 @@ task:
matrix:
- name: NetBSD - Meson
+ # See REPO_CI_AUTOMATIC_TRIGGER_TASKS in .cirrus.star
+ trigger_type: $CI_TRIGGER_TYPE_NETBSD
only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*netbsd.*'
env:
OS_NAME: netbsd
@@ -253,6 +254,8 @@ task:
<<: *netbsd_task_template
- name: OpenBSD - Meson
+ # See REPO_CI_AUTOMATIC_TRIGGER_TASKS in .cirrus.star
+ trigger_type: $CI_TRIGGER_TYPE_OPENBSD
only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*openbsd.*'
env:
OS_NAME: openbsd
@@ -706,13 +709,11 @@ task:
<< : *WINDOWS_ENVIRONMENT_BASE
name: Windows - Server 2019, MinGW64 - Meson
- # due to resource constraints we don't run this task by default for now
- trigger_type: manual
- # worth using only_if despite being manual, otherwise this task will show up
- # when e.g. ci-os-only: linux is used.
- only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*mingw.*'
- # otherwise it'll be sorted before other tasks
+ # See REPO_CI_AUTOMATIC_TRIGGER_TASKS in .cirrus.star.
+ trigger_type: $CI_TRIGGER_TYPE_MINGW
+
depends_on: SanityCheck
+ only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*mingw.*'
env:
TEST_JOBS: 4 # higher concurrency causes occasional failures
diff --git a/.cirrus.yml b/.cirrus.yml
index 33c6e481d74..3f75852e84e 100644
--- a/.cirrus.yml
+++ b/.cirrus.yml
@@ -10,12 +10,20 @@
#
# 1) the contents of this file
#
-# 2) if defined, the contents of the file referenced by the, repository
+# 2) computed environment variables
+#
+# Used to enable/disable tasks based on the execution environment. See
+# .cirrus.star: compute_environment_vars()
+#
+# 3) if defined, the contents of the file referenced by the, repository
# level, REPO_CI_CONFIG_GIT_URL variable (see
# https://cirrus-ci.org/guide/programming-tasks/#fs for the accepted
# format)
#
-# 3) .cirrus.tasks.yml
+# This allows running tasks in a different execution environment than the
+# default, e.g. to have sufficient resources for cfbot.
+#
+# 4) .cirrus.tasks.yml
#
# This composition is done by .cirrus.star
diff --git a/src/tools/ci/README b/src/tools/ci/README
index 12c1e7c308f..d183648a8d0 100644
--- a/src/tools/ci/README
+++ b/src/tools/ci/README
@@ -82,3 +82,14 @@ defined in .cirrus.yml, by redefining the relevant yaml anchors.
Custom compute resources can be provided using
- https://cirrus-ci.org/guide/supported-computing-services/
- https://cirrus-ci.org/guide/persistent-workers/
+
+
+Enabling manual tasks by default
+================================
+
+Some tasks are not triggered automatically by default, to avoid using up CI
+credits too quickly. This can be changed on the repository level, e.g. when
+custom compute resources are configured.
+
+The following repository level environment variables are recognized:
+- REPO_CI_AUTOMATIC_TRIGGER_TASKS - space-separated list of (mingw|netbsd|openbsd)
--
2.48.1
