XComp commented on code in PR #28831:
URL: https://github.com/apache/flink/pull/28831#discussion_r3702002646
##########
.github/workflows/nightly.yml:
##########
@@ -77,6 +109,163 @@ jobs:
s3_access_key: ${{ secrets.IT_CASE_S3_ACCESS_KEY }}
s3_secret_key: ${{ secrets.IT_CASE_S3_SECRET_KEY }}
+ snapshot_binary:
+ name: "Snapshot Binary Release"
+ runs-on: ubuntu-24.04
+ container:
+ image: apache/flink-ci-docker:java_8_11_17_21_25_maven_386_jammy
+ options: --init --privileged
+ env:
+ MOUNTED_WORKING_DIR: /__w/flink/flink
+ CONTAINER_LOCAL_WORKING_DIR: /root/flink
+ MAVEN_REPO_FOLDER: /root/.m2/repository
+ MAVEN_ARGS: -Dmaven.repo.local=/root/.m2/repository
+ RELEASE_DIR: /root/flink/tools/releasing/release
+ steps:
+ - name: "Flink Checkout"
+ uses: actions/checkout@v7
+ with:
+ persist-credentials: false
+ - name: "Initialize job"
+ uses: "./.github/actions/job_init"
+ with:
+ jdk_version: 11
+ maven_repo_folder: ${{ env.MAVEN_REPO_FOLDER }}
+ source_directory: ${{ env.MOUNTED_WORKING_DIR }}
+ target_directory: ${{ env.CONTAINER_LOCAL_WORKING_DIR }}
+ - name: "Build snapshot binary release"
+ working-directory: ${{ env.CONTAINER_LOCAL_WORKING_DIR }}
+ shell: bash
+ run: |
+ # errexit needs to be disabled explicitly here because
maven-utils.sh handles the error if a mirror is not available
+ set +o errexit
+ source ./tools/ci/maven-utils.sh
+ set -o errexit
+ run_mvn -version
+ export MVN="run_mvn"
+ export RELEASE_VERSION=$(MVN_RUN_VERBOSE=false run_mvn help:evaluate
-Dexpression=project.version -q -DforceStdout)
+ echo "Determined RELEASE_VERSION as '$RELEASE_VERSION'"
+ cd tools
+ MVN_RUN_VERBOSE=true SKIP_GPG=true SKIP_PYTHON_WHEELS=true
./releasing/create_binary_release.sh
+ echo "Created files:"
+ find ./releasing/release
+ cd ..
+ # Uploaded for every branch so that the binaries can be inspected and
+ # compared against the ones the Azure pipeline produces.
+ - name: "Upload snapshot binary release as build artifact"
+ uses: actions/upload-artifact@v7
+ with:
+ name: snapshot-binary-release-${{ github.run_number }}
+ path: ${{ env.RELEASE_DIR }}
+ if-no-files-found: error
+ retention-days: 5
+ - name: "Upload artifacts to S3"
+ if: ${{ contains(fromJSON(env.PUBLISH_SNAPSHOT_BRANCHES),
github.ref_name) }}
+ working-directory: ${{ env.CONTAINER_LOCAL_WORKING_DIR }}
+ shell: bash
+ run: |
+ source ./tools/ci/deploy_nightly_to_s3.sh
+ # branches in the shadow phase upload to a GHA-specific prefix, so
that
+ # they don't overwrite the builds AZP is still uploading
Review Comment:
```suggestion
# branches in the shadow phase upload to a GHA-specific prefix, so
that
# they don't overwrite the builds AZP is still uploading
```
##########
.github/workflows/nightly.yml:
##########
@@ -27,6 +27,38 @@ concurrency:
permissions: read-all
+env:
+ # Branches whose nightly snapshots are published by this pipeline (binaries
+ # to S3, jars to repository.apache.org). Branches not listed here will build
+ # snapshots, and keep them as build artifacts of the workflow run only.
+ # This allows the migration from Azure Pipelines to GitHub Actions to happen
+ # one release branch at a time, without both pipelines publishing over each
+ # other: add a branch here once its snapshots are no longer published by
+ # the Azure pipeline, or while it is in the shadow phase described under
+ # SHADOW_SNAPSHOT_BRANCHES below.
+ #
+ # Adding a branch here needs a matching change to that branch's copy of
+ # tools/azure-pipelines/build-apache-repo.yml, ideally in the same commit.
+ # Remove the "cron_snapshot_deployment" build-nightly-dist.yml template from
+ # the "cron_build" stage in tools/azure-pipelines/build-apache-repo.yml
+ # so that the Azure pipeline stops publishing snapshots for the branch.
+ # Removing that template will stop snapshots being published by AZP
+ PUBLISH_SNAPSHOT_BRANCHES: '["master"]'
+
+ # Branches in the shadow phase, where this pipeline and the Azure pipeline
both
+ # publish the same branch so that their output can be compared. Binaries for
+ # these branches are uploaded to a GHA-specific prefix in the S3 bucket
instead
+ # of the bucket root, so that the two pipelines don't overwrite each other.
+ # (Snapshot jars in Maven don't need an equivalent, as they are timestamped.)
+ #
+ # A branch belongs here only while it is *also* still being published by the
+ # Azure pipeline. Remove it from this list in the same commit that removes
the
+ # branch's "cron_snapshot_deployment" template from build-apache-repo.yml, so
+ # that it starts publishing to the bucket root as the Azure pipeline stops.
+ # Leaving a branch here after AZP has stopped means nothing is publishing its
+ # snapshots to the location that consumers read from.
+ SHADOW_SNAPSHOT_BRANCHES: '["master"]'
Review Comment:
I'm wondering whether it would be sufficient to compare binary artifacts via
the build artifacts which means we would not need to push them under a
different name at all. 🤔
##########
tools/releasing/create_binary_release.sh:
##########
@@ -21,6 +21,11 @@
## Variables with defaults (if not overwritten by environment)
##
SKIP_GPG=${SKIP_GPG:-false}
+# Set to true for automated builds (e.g. nightly snapshots) which
+# do not try to include Python wheels in the binary release.
+# Defaults to false for manual use, when the release manager will
+# first download platform wheels to flink-python/dist
Review Comment:
```suggestion
# Set to true for automated builds (e.g. nightly snapshots) which
# do not try to include Python wheels in the binary release.
# Defaults to false for manual use, when the release manager will
# first download platform wheels to flink-python/dist
```
strange spacing
##########
.github/workflows/nightly.yml:
##########
@@ -77,6 +109,163 @@ jobs:
s3_access_key: ${{ secrets.IT_CASE_S3_ACCESS_KEY }}
s3_secret_key: ${{ secrets.IT_CASE_S3_SECRET_KEY }}
+ snapshot_binary:
+ name: "Snapshot Binary Release"
+ runs-on: ubuntu-24.04
+ container:
+ image: apache/flink-ci-docker:java_8_11_17_21_25_maven_386_jammy
+ options: --init --privileged
+ env:
+ MOUNTED_WORKING_DIR: /__w/flink/flink
+ CONTAINER_LOCAL_WORKING_DIR: /root/flink
+ MAVEN_REPO_FOLDER: /root/.m2/repository
+ MAVEN_ARGS: -Dmaven.repo.local=/root/.m2/repository
+ RELEASE_DIR: /root/flink/tools/releasing/release
+ steps:
+ - name: "Flink Checkout"
+ uses: actions/checkout@v7
+ with:
+ persist-credentials: false
+ - name: "Initialize job"
+ uses: "./.github/actions/job_init"
+ with:
+ jdk_version: 11
Review Comment:
> shouldn't we rely on the default version (which is 17 now) instead of
hard-coding 11 here?
[same](https://github.com/apache/flink/pull/28831/changes#r3702086468) here
##########
.github/workflows/nightly.yml:
##########
@@ -27,6 +27,38 @@ concurrency:
permissions: read-all
+env:
+ # Branches whose nightly snapshots are published by this pipeline (binaries
+ # to S3, jars to repository.apache.org). Branches not listed here will build
+ # snapshots, and keep them as build artifacts of the workflow run only.
+ # This allows the migration from Azure Pipelines to GitHub Actions to happen
+ # one release branch at a time, without both pipelines publishing over each
+ # other: add a branch here once its snapshots are no longer published by
+ # the Azure pipeline, or while it is in the shadow phase described under
+ # SHADOW_SNAPSHOT_BRANCHES below.
+ #
+ # Adding a branch here needs a matching change to that branch's copy of
+ # tools/azure-pipelines/build-apache-repo.yml, ideally in the same commit.
+ # Remove the "cron_snapshot_deployment" build-nightly-dist.yml template from
+ # the "cron_build" stage in tools/azure-pipelines/build-apache-repo.yml
+ # so that the Azure pipeline stops publishing snapshots for the branch.
+ # Removing that template will stop snapshots being published by AZP
+ PUBLISH_SNAPSHOT_BRANCHES: '["master"]'
Review Comment:
let's not enable master until we decided to do so via mailing list - even w/
shadowing enabled.
##########
.github/workflows/nightly.yml:
##########
@@ -77,6 +109,163 @@ jobs:
s3_access_key: ${{ secrets.IT_CASE_S3_ACCESS_KEY }}
s3_secret_key: ${{ secrets.IT_CASE_S3_SECRET_KEY }}
+ snapshot_binary:
Review Comment:
I'm wondering whether we should put the snapshot jobs "behind" the CI
template runs to make use of the Maven cache instead of running it in parallel.
🤔
##########
.github/workflows/nightly.yml:
##########
@@ -77,6 +109,163 @@ jobs:
s3_access_key: ${{ secrets.IT_CASE_S3_ACCESS_KEY }}
s3_secret_key: ${{ secrets.IT_CASE_S3_SECRET_KEY }}
+ snapshot_binary:
+ name: "Snapshot Binary Release"
+ runs-on: ubuntu-24.04
+ container:
+ image: apache/flink-ci-docker:java_8_11_17_21_25_maven_386_jammy
+ options: --init --privileged
+ env:
+ MOUNTED_WORKING_DIR: /__w/flink/flink
+ CONTAINER_LOCAL_WORKING_DIR: /root/flink
+ MAVEN_REPO_FOLDER: /root/.m2/repository
+ MAVEN_ARGS: -Dmaven.repo.local=/root/.m2/repository
+ RELEASE_DIR: /root/flink/tools/releasing/release
+ steps:
+ - name: "Flink Checkout"
+ uses: actions/checkout@v7
+ with:
+ persist-credentials: false
+ - name: "Initialize job"
+ uses: "./.github/actions/job_init"
+ with:
+ jdk_version: 11
+ maven_repo_folder: ${{ env.MAVEN_REPO_FOLDER }}
+ source_directory: ${{ env.MOUNTED_WORKING_DIR }}
+ target_directory: ${{ env.CONTAINER_LOCAL_WORKING_DIR }}
+ - name: "Build snapshot binary release"
+ working-directory: ${{ env.CONTAINER_LOCAL_WORKING_DIR }}
+ shell: bash
+ run: |
+ # errexit needs to be disabled explicitly here because
maven-utils.sh handles the error if a mirror is not available
+ set +o errexit
+ source ./tools/ci/maven-utils.sh
+ set -o errexit
+ run_mvn -version
+ export MVN="run_mvn"
+ export RELEASE_VERSION=$(MVN_RUN_VERBOSE=false run_mvn help:evaluate
-Dexpression=project.version -q -DforceStdout)
+ echo "Determined RELEASE_VERSION as '$RELEASE_VERSION'"
+ cd tools
+ MVN_RUN_VERBOSE=true SKIP_GPG=true SKIP_PYTHON_WHEELS=true
./releasing/create_binary_release.sh
+ echo "Created files:"
+ find ./releasing/release
+ cd ..
+ # Uploaded for every branch so that the binaries can be inspected and
+ # compared against the ones the Azure pipeline produces.
+ - name: "Upload snapshot binary release as build artifact"
+ uses: actions/upload-artifact@v7
+ with:
+ name: snapshot-binary-release-${{ github.run_number }}
+ path: ${{ env.RELEASE_DIR }}
+ if-no-files-found: error
+ retention-days: 5
+ - name: "Upload artifacts to S3"
+ if: ${{ contains(fromJSON(env.PUBLISH_SNAPSHOT_BRANCHES),
github.ref_name) }}
+ working-directory: ${{ env.CONTAINER_LOCAL_WORKING_DIR }}
+ shell: bash
+ run: |
+ source ./tools/ci/deploy_nightly_to_s3.sh
Review Comment:
Should we guard with a warning if the secrets are missing. This is going to
be expected for a bit until we have the secrets properly set up.
##########
.github/workflows/nightly.yml:
##########
@@ -77,6 +109,163 @@ jobs:
s3_access_key: ${{ secrets.IT_CASE_S3_ACCESS_KEY }}
s3_secret_key: ${{ secrets.IT_CASE_S3_SECRET_KEY }}
+ snapshot_binary:
+ name: "Snapshot Binary Release"
+ runs-on: ubuntu-24.04
+ container:
+ image: apache/flink-ci-docker:java_8_11_17_21_25_maven_386_jammy
+ options: --init --privileged
+ env:
+ MOUNTED_WORKING_DIR: /__w/flink/flink
+ CONTAINER_LOCAL_WORKING_DIR: /root/flink
+ MAVEN_REPO_FOLDER: /root/.m2/repository
+ MAVEN_ARGS: -Dmaven.repo.local=/root/.m2/repository
+ RELEASE_DIR: /root/flink/tools/releasing/release
+ steps:
+ - name: "Flink Checkout"
+ uses: actions/checkout@v7
+ with:
+ persist-credentials: false
+ - name: "Initialize job"
+ uses: "./.github/actions/job_init"
+ with:
+ jdk_version: 11
+ maven_repo_folder: ${{ env.MAVEN_REPO_FOLDER }}
+ source_directory: ${{ env.MOUNTED_WORKING_DIR }}
+ target_directory: ${{ env.CONTAINER_LOCAL_WORKING_DIR }}
+ - name: "Build snapshot binary release"
+ working-directory: ${{ env.CONTAINER_LOCAL_WORKING_DIR }}
+ shell: bash
+ run: |
+ # errexit needs to be disabled explicitly here because
maven-utils.sh handles the error if a mirror is not available
+ set +o errexit
+ source ./tools/ci/maven-utils.sh
+ set -o errexit
+ run_mvn -version
+ export MVN="run_mvn"
+ export RELEASE_VERSION=$(MVN_RUN_VERBOSE=false run_mvn help:evaluate
-Dexpression=project.version -q -DforceStdout)
+ echo "Determined RELEASE_VERSION as '$RELEASE_VERSION'"
+ cd tools
+ MVN_RUN_VERBOSE=true SKIP_GPG=true SKIP_PYTHON_WHEELS=true
./releasing/create_binary_release.sh
+ echo "Created files:"
+ find ./releasing/release
+ cd ..
+ # Uploaded for every branch so that the binaries can be inspected and
+ # compared against the ones the Azure pipeline produces.
+ - name: "Upload snapshot binary release as build artifact"
+ uses: actions/upload-artifact@v7
+ with:
+ name: snapshot-binary-release-${{ github.run_number }}
+ path: ${{ env.RELEASE_DIR }}
+ if-no-files-found: error
+ retention-days: 5
+ - name: "Upload artifacts to S3"
+ if: ${{ contains(fromJSON(env.PUBLISH_SNAPSHOT_BRANCHES),
github.ref_name) }}
+ working-directory: ${{ env.CONTAINER_LOCAL_WORKING_DIR }}
+ shell: bash
+ run: |
+ source ./tools/ci/deploy_nightly_to_s3.sh
+ # branches in the shadow phase upload to a GHA-specific prefix, so
that
+ # they don't overwrite the builds AZP is still uploading
+ upload_to_s3 ./tools/releasing/release "${S3_PREFIX}"
+ env:
+ S3_PREFIX: ${{ contains(fromJSON(env.SHADOW_SNAPSHOT_BRANCHES),
github.ref_name) && 'gha-trial/' || '' }}
+ ARTIFACTS_S3_BUCKET: ${{ secrets.ARTIFACTS_S3_BUCKET }}
+ AWS_ACCESS_KEY_ID: ${{ secrets.ARTIFACTS_AWS_ACCESS_KEY_ID }}
+ AWS_SECRET_ACCESS_KEY: ${{ secrets.ARTIFACTS_AWS_SECRET_ACCESS_KEY }}
+
+ snapshot_maven:
+ name: "Snapshot Maven Deploy"
+ runs-on: ubuntu-24.04
+ timeout-minutes: 240
+ container:
+ image: apache/flink-ci-docker:java_8_11_17_21_25_maven_386_jammy
+ options: --init --privileged
+ env:
+ MOUNTED_WORKING_DIR: /__w/flink/flink
+ CONTAINER_LOCAL_WORKING_DIR: /root/flink
+ MAVEN_REPO_FOLDER: /root/.m2/repository
+ MAVEN_ARGS: -Dmaven.repo.local=/root/.m2/repository
+ STAGING_JARS_DIR: /root/staging-jars
+ STAGING_JARS_TARBALL: /root/staging-jars.tar.gz
+ steps:
+ - name: "Flink Checkout"
+ uses: actions/checkout@v7
+ with:
+ persist-credentials: false
+ - name: "Initialize job"
+ uses: "./.github/actions/job_init"
+ with:
+ jdk_version: 11
+ maven_repo_folder: ${{ env.MAVEN_REPO_FOLDER }}
+ source_directory: ${{ env.MOUNTED_WORKING_DIR }}
+ target_directory: ${{ env.CONTAINER_LOCAL_WORKING_DIR }}
+ - name: "Deploy Maven snapshot"
+ working-directory: ${{ env.CONTAINER_LOCAL_WORKING_DIR }}
+ shell: bash
+ run: |
+ # errexit needs to be disabled explicitly here because
maven-utils.sh handles the error if a mirror is not available
+ set +o errexit
+ source ./tools/ci/maven-utils.sh
+ set -o errexit
+ run_mvn -version
+
+ cd tools
+ cat << EOF > deploy-settings.xml
+ <settings xmlns="http://maven.apache.org/SETTINGS/1.0.0";
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
+ xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.0.0
+
https://maven.apache.org/xsd/settings-1.0.0.xsd";>
+ <servers>
+ <server>
+ <id>apache.snapshots.https</id>
+ <username>${MAVEN_DEPLOY_USER}</username>
+ <password>${MAVEN_DEPLOY_PASS}</password>
+ </server>
+ </servers>
+ <mirrors>
+ <mirror>
+ <id>google-maven-central</id>
+ <name>GCS Maven Central mirror</name>
+
<url>https://maven-central-eu.storage-download.googleapis.com/maven2/</url>
+ <mirrorOf>central</mirrorOf>
+ </mirror>
+ </mirrors>
+ </settings>
+ EOF
+
+ export CUSTOM_OPTIONS="${MVN_GLOBAL_OPTIONS_WITHOUT_MIRROR}
-Dgpg.skip -Drat.skip -Dcheckstyle.skip --settings $(pwd)/deploy-settings.xml"
+
+ if [ "${PUBLISH_SNAPSHOTS}" != "true" ]; then
+ # this branch's snapshot jars are published by the Azure pipeline,
so
+ # deploy into a local repository that is kept as a build artifact
+ # instead of pushing to repository.apache.org
+ echo "'${GITHUB_REF_NAME}' does not publish snapshots - deploying
to ${STAGING_JARS_DIR} instead of repository.apache.org"
+ export CUSTOM_OPTIONS="${CUSTOM_OPTIONS}
-DaltDeploymentRepository=staging::file://${STAGING_JARS_DIR}"
+ fi
+
+ export MVN_RUN_VERBOSE=true
+ ./releasing/deploy_staging_jars.sh
+ env:
+ MAVEN_DEPLOY_USER: ${{ secrets.MAVEN_DEPLOY_USER }}
+ MAVEN_DEPLOY_PASS: ${{ secrets.MAVEN_DEPLOY_PASS }}
+ PUBLISH_SNAPSHOTS: ${{
contains(fromJSON(env.PUBLISH_SNAPSHOT_BRANCHES), github.ref_name) }}
+ - name: "Archive locally deployed snapshot jars"
+ if: ${{ !contains(fromJSON(env.PUBLISH_SNAPSHOT_BRANCHES),
github.ref_name) }}
+ shell: bash
+ run: |
+ du -sh "${STAGING_JARS_DIR}"
+ tar --create --gzip --file "${STAGING_JARS_TARBALL}" -C
"${STAGING_JARS_DIR}" .
+ ls -lh "${STAGING_JARS_TARBALL}"
+ - name: "Upload snapshot jars as build artifact"
+ if: ${{ !contains(fromJSON(env.PUBLISH_SNAPSHOT_BRANCHES),
github.ref_name) }}
+ uses: actions/upload-artifact@v7
+ with:
+ name: snapshot-maven-jars-${{ github.run_number }}
+ path: ${{ env.STAGING_JARS_TARBALL }}
+ if-no-files-found: error
+ retention-days: 5
Review Comment:
maybe, we should double-check with INFRA whether uploading to s3 on a prefix
or build artifacts are a preferred approach. Or whether there's a 3rd option. 🤔
##########
.github/workflows/nightly.yml:
##########
@@ -77,6 +109,163 @@ jobs:
s3_access_key: ${{ secrets.IT_CASE_S3_ACCESS_KEY }}
s3_secret_key: ${{ secrets.IT_CASE_S3_SECRET_KEY }}
+ snapshot_binary:
+ name: "Snapshot Binary Release"
+ runs-on: ubuntu-24.04
+ container:
+ image: apache/flink-ci-docker:java_8_11_17_21_25_maven_386_jammy
+ options: --init --privileged
+ env:
+ MOUNTED_WORKING_DIR: /__w/flink/flink
+ CONTAINER_LOCAL_WORKING_DIR: /root/flink
+ MAVEN_REPO_FOLDER: /root/.m2/repository
+ MAVEN_ARGS: -Dmaven.repo.local=/root/.m2/repository
+ RELEASE_DIR: /root/flink/tools/releasing/release
+ steps:
+ - name: "Flink Checkout"
+ uses: actions/checkout@v7
+ with:
+ persist-credentials: false
+ - name: "Initialize job"
+ uses: "./.github/actions/job_init"
+ with:
+ jdk_version: 11
+ maven_repo_folder: ${{ env.MAVEN_REPO_FOLDER }}
+ source_directory: ${{ env.MOUNTED_WORKING_DIR }}
+ target_directory: ${{ env.CONTAINER_LOCAL_WORKING_DIR }}
+ - name: "Build snapshot binary release"
+ working-directory: ${{ env.CONTAINER_LOCAL_WORKING_DIR }}
+ shell: bash
+ run: |
+ # errexit needs to be disabled explicitly here because
maven-utils.sh handles the error if a mirror is not available
+ set +o errexit
+ source ./tools/ci/maven-utils.sh
+ set -o errexit
+ run_mvn -version
+ export MVN="run_mvn"
+ export RELEASE_VERSION=$(MVN_RUN_VERBOSE=false run_mvn help:evaluate
-Dexpression=project.version -q -DforceStdout)
+ echo "Determined RELEASE_VERSION as '$RELEASE_VERSION'"
+ cd tools
+ MVN_RUN_VERBOSE=true SKIP_GPG=true SKIP_PYTHON_WHEELS=true
./releasing/create_binary_release.sh
+ echo "Created files:"
+ find ./releasing/release
+ cd ..
+ # Uploaded for every branch so that the binaries can be inspected and
+ # compared against the ones the Azure pipeline produces.
+ - name: "Upload snapshot binary release as build artifact"
+ uses: actions/upload-artifact@v7
+ with:
+ name: snapshot-binary-release-${{ github.run_number }}
+ path: ${{ env.RELEASE_DIR }}
+ if-no-files-found: error
+ retention-days: 5
+ - name: "Upload artifacts to S3"
+ if: ${{ contains(fromJSON(env.PUBLISH_SNAPSHOT_BRANCHES),
github.ref_name) }}
+ working-directory: ${{ env.CONTAINER_LOCAL_WORKING_DIR }}
+ shell: bash
+ run: |
+ source ./tools/ci/deploy_nightly_to_s3.sh
+ # branches in the shadow phase upload to a GHA-specific prefix, so
that
+ # they don't overwrite the builds AZP is still uploading
+ upload_to_s3 ./tools/releasing/release "${S3_PREFIX}"
+ env:
+ S3_PREFIX: ${{ contains(fromJSON(env.SHADOW_SNAPSHOT_BRANCHES),
github.ref_name) && 'gha-trial/' || '' }}
+ ARTIFACTS_S3_BUCKET: ${{ secrets.ARTIFACTS_S3_BUCKET }}
+ AWS_ACCESS_KEY_ID: ${{ secrets.ARTIFACTS_AWS_ACCESS_KEY_ID }}
+ AWS_SECRET_ACCESS_KEY: ${{ secrets.ARTIFACTS_AWS_SECRET_ACCESS_KEY }}
+
+ snapshot_maven:
+ name: "Snapshot Maven Deploy"
+ runs-on: ubuntu-24.04
+ timeout-minutes: 240
+ container:
+ image: apache/flink-ci-docker:java_8_11_17_21_25_maven_386_jammy
+ options: --init --privileged
+ env:
+ MOUNTED_WORKING_DIR: /__w/flink/flink
+ CONTAINER_LOCAL_WORKING_DIR: /root/flink
+ MAVEN_REPO_FOLDER: /root/.m2/repository
+ MAVEN_ARGS: -Dmaven.repo.local=/root/.m2/repository
+ STAGING_JARS_DIR: /root/staging-jars
+ STAGING_JARS_TARBALL: /root/staging-jars.tar.gz
+ steps:
+ - name: "Flink Checkout"
+ uses: actions/checkout@v7
+ with:
+ persist-credentials: false
+ - name: "Initialize job"
+ uses: "./.github/actions/job_init"
+ with:
+ jdk_version: 11
+ maven_repo_folder: ${{ env.MAVEN_REPO_FOLDER }}
+ source_directory: ${{ env.MOUNTED_WORKING_DIR }}
+ target_directory: ${{ env.CONTAINER_LOCAL_WORKING_DIR }}
+ - name: "Deploy Maven snapshot"
+ working-directory: ${{ env.CONTAINER_LOCAL_WORKING_DIR }}
+ shell: bash
+ run: |
+ # errexit needs to be disabled explicitly here because
maven-utils.sh handles the error if a mirror is not available
+ set +o errexit
+ source ./tools/ci/maven-utils.sh
+ set -o errexit
+ run_mvn -version
+
+ cd tools
+ cat << EOF > deploy-settings.xml
+ <settings xmlns="http://maven.apache.org/SETTINGS/1.0.0";
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
+ xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.0.0
+
https://maven.apache.org/xsd/settings-1.0.0.xsd";>
+ <servers>
+ <server>
+ <id>apache.snapshots.https</id>
+ <username>${MAVEN_DEPLOY_USER}</username>
+ <password>${MAVEN_DEPLOY_PASS}</password>
+ </server>
+ </servers>
+ <mirrors>
+ <mirror>
+ <id>google-maven-central</id>
+ <name>GCS Maven Central mirror</name>
+
<url>https://maven-central-eu.storage-download.googleapis.com/maven2/</url>
+ <mirrorOf>central</mirrorOf>
+ </mirror>
+ </mirrors>
+ </settings>
+ EOF
+
+ export CUSTOM_OPTIONS="${MVN_GLOBAL_OPTIONS_WITHOUT_MIRROR}
-Dgpg.skip -Drat.skip -Dcheckstyle.skip --settings $(pwd)/deploy-settings.xml"
+
+ if [ "${PUBLISH_SNAPSHOTS}" != "true" ]; then
+ # this branch's snapshot jars are published by the Azure pipeline,
so
+ # deploy into a local repository that is kept as a build artifact
+ # instead of pushing to repository.apache.org
+ echo "'${GITHUB_REF_NAME}' does not publish snapshots - deploying
to ${STAGING_JARS_DIR} instead of repository.apache.org"
+ export CUSTOM_OPTIONS="${CUSTOM_OPTIONS}
-DaltDeploymentRepository=staging::file://${STAGING_JARS_DIR}"
+ fi
+
+ export MVN_RUN_VERBOSE=true
+ ./releasing/deploy_staging_jars.sh
+ env:
+ MAVEN_DEPLOY_USER: ${{ secrets.MAVEN_DEPLOY_USER }}
+ MAVEN_DEPLOY_PASS: ${{ secrets.MAVEN_DEPLOY_PASS }}
+ PUBLISH_SNAPSHOTS: ${{
contains(fromJSON(env.PUBLISH_SNAPSHOT_BRANCHES), github.ref_name) }}
+ - name: "Archive locally deployed snapshot jars"
+ if: ${{ !contains(fromJSON(env.PUBLISH_SNAPSHOT_BRANCHES),
github.ref_name) }}
+ shell: bash
+ run: |
+ du -sh "${STAGING_JARS_DIR}"
+ tar --create --gzip --file "${STAGING_JARS_TARBALL}" -C
"${STAGING_JARS_DIR}" .
+ ls -lh "${STAGING_JARS_TARBALL}"
+ - name: "Upload snapshot jars as build artifact"
+ if: ${{ !contains(fromJSON(env.PUBLISH_SNAPSHOT_BRANCHES),
github.ref_name) }}
+ uses: actions/upload-artifact@v7
+ with:
+ name: snapshot-maven-jars-${{ github.run_number }}
+ path: ${{ env.STAGING_JARS_TARBALL }}
+ if-no-files-found: error
+ retention-days: 5
Review Comment:
can we hide that behind a flag as well which, when set to 0 doesn't run the
artifact upload at all?
##########
.github/workflows/nightly.yml:
##########
@@ -77,6 +109,163 @@ jobs:
s3_access_key: ${{ secrets.IT_CASE_S3_ACCESS_KEY }}
s3_secret_key: ${{ secrets.IT_CASE_S3_SECRET_KEY }}
+ snapshot_binary:
+ name: "Snapshot Binary Release"
+ runs-on: ubuntu-24.04
+ container:
+ image: apache/flink-ci-docker:java_8_11_17_21_25_maven_386_jammy
+ options: --init --privileged
+ env:
+ MOUNTED_WORKING_DIR: /__w/flink/flink
+ CONTAINER_LOCAL_WORKING_DIR: /root/flink
+ MAVEN_REPO_FOLDER: /root/.m2/repository
+ MAVEN_ARGS: -Dmaven.repo.local=/root/.m2/repository
+ RELEASE_DIR: /root/flink/tools/releasing/release
+ steps:
+ - name: "Flink Checkout"
+ uses: actions/checkout@v7
+ with:
+ persist-credentials: false
+ - name: "Initialize job"
+ uses: "./.github/actions/job_init"
+ with:
+ jdk_version: 11
+ maven_repo_folder: ${{ env.MAVEN_REPO_FOLDER }}
+ source_directory: ${{ env.MOUNTED_WORKING_DIR }}
+ target_directory: ${{ env.CONTAINER_LOCAL_WORKING_DIR }}
+ - name: "Build snapshot binary release"
+ working-directory: ${{ env.CONTAINER_LOCAL_WORKING_DIR }}
+ shell: bash
+ run: |
+ # errexit needs to be disabled explicitly here because
maven-utils.sh handles the error if a mirror is not available
+ set +o errexit
+ source ./tools/ci/maven-utils.sh
+ set -o errexit
+ run_mvn -version
+ export MVN="run_mvn"
+ export RELEASE_VERSION=$(MVN_RUN_VERBOSE=false run_mvn help:evaluate
-Dexpression=project.version -q -DforceStdout)
+ echo "Determined RELEASE_VERSION as '$RELEASE_VERSION'"
+ cd tools
+ MVN_RUN_VERBOSE=true SKIP_GPG=true SKIP_PYTHON_WHEELS=true
./releasing/create_binary_release.sh
+ echo "Created files:"
+ find ./releasing/release
+ cd ..
+ # Uploaded for every branch so that the binaries can be inspected and
+ # compared against the ones the Azure pipeline produces.
+ - name: "Upload snapshot binary release as build artifact"
+ uses: actions/upload-artifact@v7
+ with:
+ name: snapshot-binary-release-${{ github.run_number }}
+ path: ${{ env.RELEASE_DIR }}
+ if-no-files-found: error
+ retention-days: 5
+ - name: "Upload artifacts to S3"
+ if: ${{ contains(fromJSON(env.PUBLISH_SNAPSHOT_BRANCHES),
github.ref_name) }}
+ working-directory: ${{ env.CONTAINER_LOCAL_WORKING_DIR }}
+ shell: bash
+ run: |
+ source ./tools/ci/deploy_nightly_to_s3.sh
+ # branches in the shadow phase upload to a GHA-specific prefix, so
that
+ # they don't overwrite the builds AZP is still uploading
+ upload_to_s3 ./tools/releasing/release "${S3_PREFIX}"
+ env:
+ S3_PREFIX: ${{ contains(fromJSON(env.SHADOW_SNAPSHOT_BRANCHES),
github.ref_name) && 'gha-trial/' || '' }}
+ ARTIFACTS_S3_BUCKET: ${{ secrets.ARTIFACTS_S3_BUCKET }}
+ AWS_ACCESS_KEY_ID: ${{ secrets.ARTIFACTS_AWS_ACCESS_KEY_ID }}
+ AWS_SECRET_ACCESS_KEY: ${{ secrets.ARTIFACTS_AWS_SECRET_ACCESS_KEY }}
+
+ snapshot_maven:
+ name: "Snapshot Maven Deploy"
+ runs-on: ubuntu-24.04
+ timeout-minutes: 240
+ container:
+ image: apache/flink-ci-docker:java_8_11_17_21_25_maven_386_jammy
+ options: --init --privileged
+ env:
+ MOUNTED_WORKING_DIR: /__w/flink/flink
+ CONTAINER_LOCAL_WORKING_DIR: /root/flink
+ MAVEN_REPO_FOLDER: /root/.m2/repository
+ MAVEN_ARGS: -Dmaven.repo.local=/root/.m2/repository
+ STAGING_JARS_DIR: /root/staging-jars
+ STAGING_JARS_TARBALL: /root/staging-jars.tar.gz
+ steps:
+ - name: "Flink Checkout"
+ uses: actions/checkout@v7
+ with:
+ persist-credentials: false
+ - name: "Initialize job"
+ uses: "./.github/actions/job_init"
+ with:
+ jdk_version: 11
Review Comment:
shouldn't we rely on the default version (which is 17 now) instead of
hard-coding 11 here?
##########
.github/workflows/nightly.yml:
##########
@@ -77,6 +109,163 @@ jobs:
s3_access_key: ${{ secrets.IT_CASE_S3_ACCESS_KEY }}
s3_secret_key: ${{ secrets.IT_CASE_S3_SECRET_KEY }}
+ snapshot_binary:
+ name: "Snapshot Binary Release"
+ runs-on: ubuntu-24.04
+ container:
+ image: apache/flink-ci-docker:java_8_11_17_21_25_maven_386_jammy
+ options: --init --privileged
+ env:
+ MOUNTED_WORKING_DIR: /__w/flink/flink
+ CONTAINER_LOCAL_WORKING_DIR: /root/flink
+ MAVEN_REPO_FOLDER: /root/.m2/repository
+ MAVEN_ARGS: -Dmaven.repo.local=/root/.m2/repository
+ RELEASE_DIR: /root/flink/tools/releasing/release
+ steps:
+ - name: "Flink Checkout"
+ uses: actions/checkout@v7
+ with:
+ persist-credentials: false
+ - name: "Initialize job"
+ uses: "./.github/actions/job_init"
+ with:
+ jdk_version: 11
+ maven_repo_folder: ${{ env.MAVEN_REPO_FOLDER }}
+ source_directory: ${{ env.MOUNTED_WORKING_DIR }}
+ target_directory: ${{ env.CONTAINER_LOCAL_WORKING_DIR }}
+ - name: "Build snapshot binary release"
+ working-directory: ${{ env.CONTAINER_LOCAL_WORKING_DIR }}
+ shell: bash
+ run: |
+ # errexit needs to be disabled explicitly here because
maven-utils.sh handles the error if a mirror is not available
+ set +o errexit
+ source ./tools/ci/maven-utils.sh
+ set -o errexit
+ run_mvn -version
+ export MVN="run_mvn"
+ export RELEASE_VERSION=$(MVN_RUN_VERBOSE=false run_mvn help:evaluate
-Dexpression=project.version -q -DforceStdout)
+ echo "Determined RELEASE_VERSION as '$RELEASE_VERSION'"
+ cd tools
+ MVN_RUN_VERBOSE=true SKIP_GPG=true SKIP_PYTHON_WHEELS=true
./releasing/create_binary_release.sh
Review Comment:
Why don't we need an explicit echo here in case the script fails? And would
it be worth adding the echo to the script instead of changing the azure
pipeline?
##########
.github/workflows/nightly.yml:
##########
@@ -27,6 +27,38 @@ concurrency:
permissions: read-all
+env:
+ # Branches whose nightly snapshots are published by this pipeline (binaries
+ # to S3, jars to repository.apache.org). Branches not listed here will build
+ # snapshots, and keep them as build artifacts of the workflow run only.
+ # This allows the migration from Azure Pipelines to GitHub Actions to happen
+ # one release branch at a time, without both pipelines publishing over each
+ # other: add a branch here once its snapshots are no longer published by
+ # the Azure pipeline, or while it is in the shadow phase described under
+ # SHADOW_SNAPSHOT_BRANCHES below.
+ #
+ # Adding a branch here needs a matching change to that branch's copy of
+ # tools/azure-pipelines/build-apache-repo.yml, ideally in the same commit.
+ # Remove the "cron_snapshot_deployment" build-nightly-dist.yml template from
+ # the "cron_build" stage in tools/azure-pipelines/build-apache-repo.yml
+ # so that the Azure pipeline stops publishing snapshots for the branch.
+ # Removing that template will stop snapshots being published by AZP
+ PUBLISH_SNAPSHOT_BRANCHES: '["master"]'
+
+ # Branches in the shadow phase, where this pipeline and the Azure pipeline
both
+ # publish the same branch so that their output can be compared. Binaries for
Review Comment:
did we check the s3 branch whether that's valid and wouldn't cause other
problems?
##########
.github/workflows/nightly.yml:
##########
@@ -77,6 +109,163 @@ jobs:
s3_access_key: ${{ secrets.IT_CASE_S3_ACCESS_KEY }}
s3_secret_key: ${{ secrets.IT_CASE_S3_SECRET_KEY }}
+ snapshot_binary:
+ name: "Snapshot Binary Release"
+ runs-on: ubuntu-24.04
+ container:
+ image: apache/flink-ci-docker:java_8_11_17_21_25_maven_386_jammy
+ options: --init --privileged
+ env:
+ MOUNTED_WORKING_DIR: /__w/flink/flink
+ CONTAINER_LOCAL_WORKING_DIR: /root/flink
+ MAVEN_REPO_FOLDER: /root/.m2/repository
+ MAVEN_ARGS: -Dmaven.repo.local=/root/.m2/repository
+ RELEASE_DIR: /root/flink/tools/releasing/release
+ steps:
+ - name: "Flink Checkout"
+ uses: actions/checkout@v7
+ with:
+ persist-credentials: false
+ - name: "Initialize job"
+ uses: "./.github/actions/job_init"
+ with:
+ jdk_version: 11
+ maven_repo_folder: ${{ env.MAVEN_REPO_FOLDER }}
+ source_directory: ${{ env.MOUNTED_WORKING_DIR }}
+ target_directory: ${{ env.CONTAINER_LOCAL_WORKING_DIR }}
+ - name: "Build snapshot binary release"
+ working-directory: ${{ env.CONTAINER_LOCAL_WORKING_DIR }}
+ shell: bash
+ run: |
+ # errexit needs to be disabled explicitly here because
maven-utils.sh handles the error if a mirror is not available
+ set +o errexit
+ source ./tools/ci/maven-utils.sh
+ set -o errexit
+ run_mvn -version
+ export MVN="run_mvn"
+ export RELEASE_VERSION=$(MVN_RUN_VERBOSE=false run_mvn help:evaluate
-Dexpression=project.version -q -DforceStdout)
+ echo "Determined RELEASE_VERSION as '$RELEASE_VERSION'"
+ cd tools
+ MVN_RUN_VERBOSE=true SKIP_GPG=true SKIP_PYTHON_WHEELS=true
./releasing/create_binary_release.sh
+ echo "Created files:"
+ find ./releasing/release
+ cd ..
+ # Uploaded for every branch so that the binaries can be inspected and
+ # compared against the ones the Azure pipeline produces.
+ - name: "Upload snapshot binary release as build artifact"
+ uses: actions/upload-artifact@v7
+ with:
+ name: snapshot-binary-release-${{ github.run_number }}
+ path: ${{ env.RELEASE_DIR }}
+ if-no-files-found: error
+ retention-days: 5
+ - name: "Upload artifacts to S3"
+ if: ${{ contains(fromJSON(env.PUBLISH_SNAPSHOT_BRANCHES),
github.ref_name) }}
+ working-directory: ${{ env.CONTAINER_LOCAL_WORKING_DIR }}
+ shell: bash
+ run: |
+ source ./tools/ci/deploy_nightly_to_s3.sh
+ # branches in the shadow phase upload to a GHA-specific prefix, so
that
+ # they don't overwrite the builds AZP is still uploading
+ upload_to_s3 ./tools/releasing/release "${S3_PREFIX}"
+ env:
+ S3_PREFIX: ${{ contains(fromJSON(env.SHADOW_SNAPSHOT_BRANCHES),
github.ref_name) && 'gha-trial/' || '' }}
+ ARTIFACTS_S3_BUCKET: ${{ secrets.ARTIFACTS_S3_BUCKET }}
+ AWS_ACCESS_KEY_ID: ${{ secrets.ARTIFACTS_AWS_ACCESS_KEY_ID }}
+ AWS_SECRET_ACCESS_KEY: ${{ secrets.ARTIFACTS_AWS_SECRET_ACCESS_KEY }}
+
+ snapshot_maven:
+ name: "Snapshot Maven Deploy"
+ runs-on: ubuntu-24.04
+ timeout-minutes: 240
+ container:
+ image: apache/flink-ci-docker:java_8_11_17_21_25_maven_386_jammy
+ options: --init --privileged
+ env:
+ MOUNTED_WORKING_DIR: /__w/flink/flink
+ CONTAINER_LOCAL_WORKING_DIR: /root/flink
+ MAVEN_REPO_FOLDER: /root/.m2/repository
+ MAVEN_ARGS: -Dmaven.repo.local=/root/.m2/repository
+ STAGING_JARS_DIR: /root/staging-jars
+ STAGING_JARS_TARBALL: /root/staging-jars.tar.gz
+ steps:
+ - name: "Flink Checkout"
+ uses: actions/checkout@v7
+ with:
+ persist-credentials: false
+ - name: "Initialize job"
+ uses: "./.github/actions/job_init"
+ with:
+ jdk_version: 11
+ maven_repo_folder: ${{ env.MAVEN_REPO_FOLDER }}
+ source_directory: ${{ env.MOUNTED_WORKING_DIR }}
+ target_directory: ${{ env.CONTAINER_LOCAL_WORKING_DIR }}
+ - name: "Deploy Maven snapshot"
+ working-directory: ${{ env.CONTAINER_LOCAL_WORKING_DIR }}
+ shell: bash
+ run: |
+ # errexit needs to be disabled explicitly here because
maven-utils.sh handles the error if a mirror is not available
+ set +o errexit
+ source ./tools/ci/maven-utils.sh
+ set -o errexit
+ run_mvn -version
+
+ cd tools
+ cat << EOF > deploy-settings.xml
Review Comment:
Should we move this into its own script as well?
##########
.github/workflows/nightly.yml:
##########
@@ -77,6 +109,163 @@ jobs:
s3_access_key: ${{ secrets.IT_CASE_S3_ACCESS_KEY }}
s3_secret_key: ${{ secrets.IT_CASE_S3_SECRET_KEY }}
+ snapshot_binary:
+ name: "Snapshot Binary Release"
+ runs-on: ubuntu-24.04
+ container:
+ image: apache/flink-ci-docker:java_8_11_17_21_25_maven_386_jammy
+ options: --init --privileged
+ env:
+ MOUNTED_WORKING_DIR: /__w/flink/flink
+ CONTAINER_LOCAL_WORKING_DIR: /root/flink
+ MAVEN_REPO_FOLDER: /root/.m2/repository
+ MAVEN_ARGS: -Dmaven.repo.local=/root/.m2/repository
+ RELEASE_DIR: /root/flink/tools/releasing/release
Review Comment:
Do we have to set this here? `create_binary_release.sh` already sets this
variable based on the `FLINK_DIR` variable
##########
.github/workflows/nightly.yml:
##########
@@ -27,6 +27,38 @@ concurrency:
permissions: read-all
+env:
+ # Branches whose nightly snapshots are published by this pipeline (binaries
Review Comment:
We might have to rewrite the comment.
https://github.com/apache/flink/blob/master/.github/workflows/nightly-trigger.yml#L84
picks the jobs pipeline yaml from the branch itself. Hence, if we don't
backport this change, older release branches wouldn't be affected anyway.
It's still good to have a guard though because we don't want to trigger
uploads on other branches through a manual trigger of the nightly workflow.
##########
.github/workflows/nightly.yml:
##########
@@ -77,6 +109,163 @@ jobs:
s3_access_key: ${{ secrets.IT_CASE_S3_ACCESS_KEY }}
s3_secret_key: ${{ secrets.IT_CASE_S3_SECRET_KEY }}
+ snapshot_binary:
Review Comment:
that said; AzureCI is not doing it either. Hence, we could also move this
into a follow-up optimization Jira subtask
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
