dalelane opened a new pull request, #28831:
URL: https://github.com/apache/flink/pull/28831
## What is the purpose of the change
The nightly Azure pipeline publishes two things that the GitHub Actions
nightly
workflow currently doesn't: release binaries uploaded to S3, and snapshot
jars
deployed to repository.apache.org. That gap needs closing before Flink can
move off
Azure Pipelines (FLINK-27075).
Which pipeline publishes a given branch will be configuration held in the
GHA nightly
workflow itself.
Because that workflow is versioned per branch, each branch can be switched
over
independently, and a branch that hasn't been switched over yet still builds
its
snapshots on GHA (keeping them as build artifacts of the workflow run rather
than
publishing them).
The aim of this is to make sure that the new GHA jobs get exercised on every
branch
before they become the canonical source of anything we publish.
`master` is deliberately listed in *both* of the lists described below,
which is the
first phase of the rollout proposed on FLINK-40248: GHA publishes master's
binaries
under a `gha-trial/` prefix while Azure continues to publish to the bucket
root, so the
two can be compared over a series of nightly runs before anything depends on
GHA's
output.
Removing master from the shadow list and removing Azure's
`cron_snapshot_deployment`
job will be a follow-up, once the outputs have been confirmed to match.
## Brief change log
- Adds `snapshot_binary` and `snapshot_maven` jobs to
`.github/workflows/nightly.yml`,
mirroring the `cron_snapshot_deployment_binary` and
`cron_snapshot_deployment_maven`
jobs in `tools/azure-pipelines/build-nightly-dist.yml`. Both run on JDK
11, the
minimum supported version, matching the Azure jobs.
- `PUBLISH_SNAPSHOT_BRANCHES` lists branches whose snapshots this pipeline
publishes.
Because `nightly.yml` is versioned per branch, this is per-branch
configuration,
and a branch that has the change backported without being added to the
list defaults
closed: it builds, but publishes nothing.
- `SHADOW_SNAPSHOT_BRANCHES` lists branches that GHA and Azure are both
publishing
during comparison. Binaries for those branches are uploaded to a
`gha-trial/` prefix
in the S3 bucket rather than the bucket root, so the two pipelines can't
overwrite
each other. Snapshot jars uploaded to Maven need no equivalent renaming,
as they
already have timestamped names.
- The binary release is uploaded as a GHA build artifact on every branch,
whether or
not it publishes, so its contents can be compared against what Azure
produces.
- When a branch isn't publishing, the Maven job deploys into a local
repository via
`-DaltDeploymentRepository` and uploads that as a tarball artifact,
instead of
deploying to repository.apache.org.
- `tools/ci/deploy_nightly_to_s3.sh`: `upload_to_s3` takes an optional
destination
prefix, defaulting to the existing behaviour of writing to the bucket
root.
- `tools/releasing/create_binary_release.sh`: adds `SKIP_PYTHON_WHEELS`,
defaulting to
`false` so the manual release process is unaffected. Nightly builds set
it to
`true`, so a binary release can be produced without the 12 platform
wheels that a
release manager downloads by hand beforehand. Also enables `nullglob`
around the
wheel loop, so it doesn't iterate over a literal `*.whl` when no wheels
are present.
- `tools/azure-pipelines/build-nightly-dist.yml`: sets
`SKIP_PYTHON_WHEELS=true`, and
now checks the exit code of `create_binary_release.sh`. Previously the
nightly Azure
build depended on the missing-wheels check failing, with that failure
masked by the
`echo`/`find` commands following it in the same inline script — which
meant any
other error in the script was silently ignored too.
## Verifying this change
This change is a CI configuration change without test coverage. I'll verify
by
running affected pipelines on my fork.
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): no
- The public API, i.e., is any changed class annotated with
`@Public(Evolving)`: no
- The serializers: no
- The runtime per-record code paths (performance sensitive): no
- Anything that affects deployment or recovery: JobManager (and its
components),
Checkpointing, Kubernetes/Yarn, ZooKeeper: no
- The S3 file system connector: no
## Documentation
- Does this pull request introduce a new feature? no
- If yes, how is the feature documented? not applicable
---
##### Was generative AI tooling used to co-author this PR?
- [X] Yes (please specify the tool below)
Generated-by: Claude Sonnet 5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
