This is an automated email from the ASF dual-hosted git repository.
hello-stephen pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 1f241cf9b45 [chore](ci) Connect PR checks and comment triggers to
current TeamCity (#70459)
1f241cf9b45 is described below
commit 1f241cf9b451730d06fc2edbba89bef27ba4efda
Author: Dongyang Li <[email protected]>
AuthorDate: Wed Nov 11 16:39:10 2026 +0800
[chore](ci) Connect PR checks and comment triggers to current TeamCity
(#70459)
### Background
Doris PR comments use the existing workflow and shell helper to select
and trigger TeamCity pipelines. This migration replaces the retired
environment and embedded credentials with the current TeamCity and a
dedicated token.
### What problem does this PR solve?
Issue Number: None
Related PR: None
PR Type: Build/CI/Chore
Problem Summary:
Connect master, branch-4.2 and branch-4.1 PR checks and comment commands
to the current TeamCity. Preserve file-based trigger/skip selection and
authorized `skip buildall`. Keep the comment entry point simple: submit
the requested pipeline with PR number, target branch and an exact VCS
revision; artifact resolution remains the target pipeline's
responsibility.
### Details
- **Change map:** The existing shell helper is 210 lines; tests cover
direct submission, command parsing, file filters, skip status coverage,
authorization and credential tracing. The workflow retains the original
file-selection functions; `.asf.yaml` updates required contexts for
three branches. No separate production Python client or new
dependencies.
- **Affected:** Compile, FEUT, BEUT, CloudUT, P0, CloudP0, VaultP0 and
NonConcurrent remain required. External and Performance are optional.
Current command keywords match the `TeamCity/<keyword>` status suffix.
- **Developer and release impact:** `run p0` submits P0 directly, with
only `env.PR_NUMBER` and `env.CI_BRANCH` as business parameters and the
commit in TeamCity's native revision field. The client does not search
or convert artifact manifests, choose producers, automatically compile
missing artifacts, override artifact dependencies or deduplicate
requests. TeamCity owns artifact retrieval and queue replacement. `run
buildall` and the existing authorized `skip buildall` policy remain;
Performance is enabled only for master using a positive branch
condition. Temporary validation VCS roots are rejected before
submission.
- **File selection:** Existing FEUT, BEUT, CloudUT, regression and
Performance file-pattern functions remain unchanged.
`trigger_or_skip_build` submits related changes and publishes skip
statuses for unrelated changes. Skipping Compile also resolves its five
downstream checks. Failed file lookups do not falsely select a skip.
- **Behavior change:** The new environment publishes current check
names; External and Performance no longer block merges. Commands run
trusted default-branch code. Token authentication replaces the retired
endpoint and public account credentials; authenticated requests are not
traced.
- **License and third-party:** Existing Apache licensing; no dependency
additions.
- **Rollback:** Revert workflow routing and required-context changes
together without restoring retired credentials or interrupting running
builds.
### Testing
- Local: 26 tests pass, including direct submission of all five
regression lanes with only two business parameters; exact revision
binding; repeated submissions; file filtering; skip buildall permissions
and status coverage on master and both release branches; stale heads;
HTTP failures; and token redaction.
- Local: actionlint, shell syntax validation and git diff --check pass.
- Live: Previous CloudUT submission through the existing shell helper
completed successfully on branch-4.2 (764 passed, 27 ignored). Earlier
Compile canaries completed on all three branches. These results predate
this final simplification.
- Live: Final direct P0 submission
[1065](https://8.218.164.131/buildConfiguration/Doris_P0/1065) retained
the exact requested commit and did not create a Compile dependency. It
failed before execution because the target pipeline still selects build
number `UNBOUND`; commit-based artifact lookup is not implemented there
yet. Compile 852 has an unexpired artifact manifest for this exact
PR/head, so this failure is a pipeline lookup gap rather than missing
compilation. All five regression configurations currently use the same
UNBOUND default. CloudUT canaries
[1066](https://8.218.164.131/buildConfiguration/Doris_CloudUT/1066)
(master),
[1067](https://8.218.164.131/buildConfiguration/Doris_CloudUT/1067)
(branch-4.2), and
[1068](https://8.218.164.131/buildConfiguration/Doris_CloudUT/1068)
(branch-4.1) are running. The live harness uses the committed shell
helper and substitutes server-local authentication/PR-read transport; it
does not post GitHub comments.
### Notes for Reviewers
The issue_comment workflow becomes active only after merge into the
default branch. Live branch protection must be checked after ASF
reconciliation. Regression pipelines must resolve artifacts for the
requested commit and fail when unavailable; the comment client
deliberately has no producer fallback. The experimental commit-tag
approach is not part of this PR and has not been deployed to production
regression configurations. Performance's temporary validation VCS
configuration still prevents general PR comment runs.
---
.asf.yaml | 54 ++-
.github/workflows/comment-to-trigger-teamcity.yml | 267 ++++----------
regression-test/pipeline/common/github-utils.sh | 4 +-
regression-test/pipeline/common/teamcity-utils.sh | 399 ++++++---------------
.../pipeline/common/tests/test_teamcity_utils.py | 352 ++++++++++++++++++
5 files changed, 557 insertions(+), 519 deletions(-)
diff --git a/.asf.yaml b/.asf.yaml
index ca9e5e0c381..bd0f6df1571 100644
--- a/.asf.yaml
+++ b/.asf.yaml
@@ -42,6 +42,8 @@ github:
merge: false
rebase: false
protected_branches:
+ # master, branch-4.2 and branch-4.1 use the migrated TeamCity contexts.
+ # External and Performance report results without gating these branches.
master:
required_status_checks:
strict: false
@@ -50,20 +52,18 @@ github:
- Check for secrets
- Clang Formatter
- CheckStyle
- - P0 Regression (Doris Regression)
- - NonConcurrent Regression (nonConcurrent)
- - External Regression (Doris External Regression)
- - cloud_p0 (Doris Cloud Regression)
- - vault_p0 (Doris Cloud Regression)
- - FE UT (Doris FE UT)
- - BE UT (Doris BE UT)
+ - TeamCity/p0
+ - TeamCity/nonconcurrent
+ - TeamCity/cloud_p0
+ - TeamCity/vault_p0
+ - TeamCity/feut
+ - TeamCity/beut
- Build Third Party Libraries (Linux)
- Build Third Party Libraries (macOS)
- Build Third Party Libraries (macOS-arm64)
- - COMPILE (DORIS_COMPILE)
+ - TeamCity/compile
- code-review
- - Cloud UT (Doris Cloud UT)
- - performance (Doris Performance)
+ - TeamCity/cloudut
required_pull_request_reviews:
dismiss_stale_reviews: true
require_code_owner_reviews: true
@@ -76,15 +76,14 @@ github:
- Clang Formatter
- CheckStyle
- Build Broker
- - FE UT (Doris FE UT)
- - BE UT (Doris BE UT)
- - Cloud UT (Doris Cloud UT)
- - COMPILE (DORIS_COMPILE)
- - P0 Regression (Doris Regression)
- - NonConcurrent Regression (nonConcurrent)
- - External Regression (Doris External Regression)
- - cloud_p0 (Doris Cloud Regression)
- - vault_p0 (Doris Cloud Regression)
+ - TeamCity/feut
+ - TeamCity/beut
+ - TeamCity/cloudut
+ - TeamCity/compile
+ - TeamCity/p0
+ - TeamCity/nonconcurrent
+ - TeamCity/cloud_p0
+ - TeamCity/vault_p0
required_pull_request_reviews:
require_code_owner_reviews: true
required_approving_review_count: 1
@@ -97,15 +96,14 @@ github:
- Clang Formatter
- CheckStyle
- Build Broker
- - FE UT (Doris FE UT)
- - BE UT (Doris BE UT)
- - Cloud UT (Doris Cloud UT)
- - COMPILE (DORIS_COMPILE)
- - P0 Regression (Doris Regression)
- - NonConcurrent Regression (nonConcurrent)
- - External Regression (Doris External Regression)
- - cloud_p0 (Doris Cloud Regression)
- - vault_p0 (Doris Cloud Regression)
+ - TeamCity/feut
+ - TeamCity/beut
+ - TeamCity/cloudut
+ - TeamCity/compile
+ - TeamCity/p0
+ - TeamCity/nonconcurrent
+ - TeamCity/cloud_p0
+ - TeamCity/vault_p0
required_pull_request_reviews:
require_code_owner_reviews: true
required_approving_review_count: 1
diff --git a/.github/workflows/comment-to-trigger-teamcity.yml
b/.github/workflows/comment-to-trigger-teamcity.yml
index 853aa6bf17f..00fa548b9b5 100644
--- a/.github/workflows/comment-to-trigger-teamcity.yml
+++ b/.github/workflows/comment-to-trigger-teamcity.yml
@@ -42,47 +42,21 @@ jobs:
contains(github.event.comment.body, 'run feut') ||
contains(github.event.comment.body, 'run cloudut') ||
contains(github.event.comment.body, 'run p0') ||
- contains(github.event.comment.body, 'run p1') ||
contains(github.event.comment.body, 'run external') ||
contains(github.event.comment.body, 'run cloud_p0') ||
- contains(github.event.comment.body, 'run cloud_p1') ||
contains(github.event.comment.body, 'run vault_p0') ||
- contains(github.event.comment.body, 'run nonConcurrent') ||
- contains(github.event.comment.body, 'run check_coverage') ||
+ contains(github.event.comment.body, 'run nonconcurrent') ||
contains(github.event.comment.body, 'run performance') ||
- contains(github.event.comment.body, 'skip buildall') ||
- contains(github.event.comment.body, 'skip check_coverage')
+ contains(github.event.comment.body, 'skip buildall')
)
}}
- # Serialize duplicate commands for one PR. The next run then sees the build
- # queued by the previous run and can safely skip an identical PR revision.
- concurrency:
- group: >-
- comment-to-trigger-teamcity-${{ github.event.issue.number }}-${{
- contains(github.event.comment.body, 'run buildall') && 'buildall' ||
- contains(github.event.comment.body, 'run compile') && 'compile' ||
- contains(github.event.comment.body, 'run beut') && 'beut' ||
- contains(github.event.comment.body, 'run feut') && 'feut' ||
- contains(github.event.comment.body, 'run cloudut') && 'cloudut' ||
- contains(github.event.comment.body, 'run p0') && 'p0' ||
- contains(github.event.comment.body, 'run p1') && 'p1' ||
- contains(github.event.comment.body, 'run external') && 'external' ||
- contains(github.event.comment.body, 'run cloud_p0') && 'cloud-p0' ||
- contains(github.event.comment.body, 'run cloud_p1') && 'cloud-p1' ||
- contains(github.event.comment.body, 'run vault_p0') && 'vault-p0' ||
- contains(github.event.comment.body, 'run nonConcurrent') &&
'non-concurrent' ||
- contains(github.event.comment.body, 'run check_coverage') &&
'check-coverage' ||
- contains(github.event.comment.body, 'run performance') &&
'performance' ||
- contains(github.event.comment.body, 'skip buildall') &&
'skip-buildall' ||
- 'skip-check-coverage'
- }}
- cancel-in-progress: false
-
runs-on: ubuntu-latest
env:
COMMENT_BODY: ${{ github.event.comment.body }}
COMMENT_USER_ID: ${{ github.event.comment.user.id }}
+ TEAMCITY_CURRENT_URL: ${{ vars.TEAMCITY_CURRENT_URL }}
+ TEAMCITY_CURRENT_TOKEN: ${{ secrets.TEAMCITY_CURRENT_TOKEN }}
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
steps:
@@ -91,49 +65,28 @@ jobs:
run: |
COMMENT_BODY=$(echo "${COMMENT_BODY}" | xargs)
PULL_REQUEST_NUM="$(echo "${{ github.event.issue.pull_request.url }}"
| awk -F/ '{print $NF}')"
- COMMIT_ID_FROM_TRIGGER="$(curl -s -H "Authorization:Bearer ${{
secrets.GITHUB_TOKEN }}" "https://api.github.com/repos/${{ github.repository
}}/pulls/${PULL_REQUEST_NUM}" | jq -r '.head.sha')"
- TARGET_BRANCH="$(curl -s -H "Authorization:Bearer ${{
secrets.GITHUB_TOKEN }}" "https://api.github.com/repos/${{ github.repository
}}/pulls/${PULL_REQUEST_NUM}" | jq -r '.base.ref')"
+ COMMIT_ID_FROM_TRIGGER="$(curl -s -H "Authorization:Bearer
${GITHUB_TOKEN}" "https://api.github.com/repos/${{ github.repository
}}/pulls/${PULL_REQUEST_NUM}" | jq -r '.head.sha')"
+ TARGET_BRANCH="$(curl -s -H "Authorization:Bearer ${GITHUB_TOKEN}"
"https://api.github.com/repos/${{ github.repository
}}/pulls/${PULL_REQUEST_NUM}" | jq -r '.base.ref')"
if [[ "${COMMENT_BODY}" == *'run buildall'* ||
"${COMMENT_BODY}" == *'run compile'* ||
"${COMMENT_BODY}" == *'run beut'* ||
"${COMMENT_BODY}" == *'run feut'* ||
"${COMMENT_BODY}" == *'run cloudut'* ||
"${COMMENT_BODY}" == *'run p0'* ||
- "${COMMENT_BODY}" == *'run p1'* ||
"${COMMENT_BODY}" == *'run external'* ||
"${COMMENT_BODY}" == *'run cloud_p0'* ||
- "${COMMENT_BODY}" == *'run cloud_p1'* ||
"${COMMENT_BODY}" == *'run vault_p0'* ||
- "${COMMENT_BODY}" == *'run nonConcurrent'* ||
- "${COMMENT_BODY}" == *'run check_coverage'* ||
+ "${COMMENT_BODY}" == *'run nonconcurrent'* ||
"${COMMENT_BODY}" == *'run performance'* ]]; then
echo "comment_trigger=true" | tee -a "$GITHUB_OUTPUT"
echo "comment_skip=false" | tee -a "$GITHUB_OUTPUT"
- echo "comment_skip_coverage=false" | tee -a "$GITHUB_OUTPUT"
elif [[ "${COMMENT_BODY}" == *'skip buildall'* ]]; then
- echo "comment_skip_coverage=false" | tee -a "$GITHUB_OUTPUT"
if [[ "${COMMENT_USER_ID}" == '27881198' ||
"${COMMENT_USER_ID}" == '2899462' ||
"${COMMENT_USER_ID}" == '37901441' ]]; then
echo "comment_trigger=false" | tee -a "$GITHUB_OUTPUT"
echo "comment_skip=true" | tee -a "$GITHUB_OUTPUT"
echo "COMMENT_USER_ID ${COMMENT_USER_ID} is allowed to skip
buildall."
- elif [[ "${COMMENT_USER_ID}" == '9208457' && "${TARGET_BRANCH}" ==
*'branch-2.1'* ]]; then
- echo "COMMENT_USER_ID ${COMMENT_USER_ID} is allowed to skip
buildall for branch-2.1"
- echo "comment_trigger=false" | tee -a "$GITHUB_OUTPUT"
- echo "comment_skip=true" | tee -a "$GITHUB_OUTPUT"
- elif [[ "${COMMENT_USER_ID}" == '98214048' && "${TARGET_BRANCH}"
== *'branch-3.0'* ]]; then
- echo "COMMENT_USER_ID ${COMMENT_USER_ID} is allowed to skip
buildall for branch-3.0"
- echo "comment_trigger=false" | tee -a "$GITHUB_OUTPUT"
- echo "comment_skip=true" | tee -a "$GITHUB_OUTPUT"
- elif [[ "${COMMENT_USER_ID}" == '101034200' && "${TARGET_BRANCH}"
== *'branch-3.1'* ]]; then
- echo "COMMENT_USER_ID ${COMMENT_USER_ID} is allowed to skip
buildall for branch-3.1"
- echo "comment_trigger=false" | tee -a "$GITHUB_OUTPUT"
- echo "comment_skip=true" | tee -a "$GITHUB_OUTPUT"
- elif [[ ("${COMMENT_USER_ID}" == '9208457' || "${COMMENT_USER_ID}"
== '2899462') && "${TARGET_BRANCH}" == *'branch-4.0'* ]]; then
- echo "COMMENT_USER_ID ${COMMENT_USER_ID} is allowed to skip
buildall for branch-4.0"
- echo "comment_trigger=false" | tee -a "$GITHUB_OUTPUT"
- echo "comment_skip=true" | tee -a "$GITHUB_OUTPUT"
elif [[ "${COMMENT_USER_ID}" == '9208457' && "${TARGET_BRANCH}" ==
*'branch-4.1'* ]]; then
echo "COMMENT_USER_ID ${COMMENT_USER_ID} is allowed to skip
buildall for branch-4.1"
echo "comment_trigger=false" | tee -a "$GITHUB_OUTPUT"
@@ -146,27 +99,9 @@ jobs:
echo "COMMENT_USER_ID ${COMMENT_USER_ID} is not allowed to
skip buildall."
exit
fi
- elif [[ "${COMMENT_BODY}" == *'skip check_coverage'* ]] &&
- [[ "${COMMENT_USER_ID}" == '37901441' ||
- "${COMMENT_USER_ID}" == '27881198' ||
- "${COMMENT_USER_ID}" == '9208457' ||
- "${COMMENT_USER_ID}" == '10553413' ||
- "${COMMENT_USER_ID}" == '1736049' ||
- "${COMMENT_USER_ID}" == '64513324' ||
- "${COMMENT_USER_ID}" == '2274896' ||
- "${COMMENT_USER_ID}" == '98214048' ||
- "${COMMENT_USER_ID}" == '19986811' ||
- "${COMMENT_USER_ID}" == '48427519' ||
- "${COMMENT_USER_ID}" == '6919662' ||
- "${COMMENT_USER_ID}" == '101034200' ||
- "${COMMENT_USER_ID}" == '2899462' ]]; then
- echo "comment_trigger=false" | tee -a "$GITHUB_OUTPUT"
- echo "comment_skip=false" | tee -a "$GITHUB_OUTPUT"
- echo "comment_skip_coverage=true" | tee -a "$GITHUB_OUTPUT"
else
echo "comment_trigger=false" | tee -a "$GITHUB_OUTPUT"
echo "comment_skip=false" | tee -a "$GITHUB_OUTPUT"
- echo "comment_skip_coverage=false" | tee -a "$GITHUB_OUTPUT"
echo "find no keyword in comment body, skip this action."
exit
fi
@@ -174,6 +109,20 @@ jobs:
echo "PULL_REQUEST_NUM=${PULL_REQUEST_NUM}" | tee -a "$GITHUB_OUTPUT"
echo "COMMIT_ID_FROM_TRIGGER=${COMMIT_ID_FROM_TRIGGER}" | tee -a
"$GITHUB_OUTPUT"
echo "TARGET_BRANCH=${TARGET_BRANCH}" | tee -a "$GITHUB_OUTPUT"
+ echo "TARGET_BRANCH=${TARGET_BRANCH}" >> "$GITHUB_ENV"
+ case "$TARGET_BRANCH" in
+ master|branch-4.2|branch-4.1)
+ if [[ "$COMMENT_BODY" == *'run '* ]]; then
+ permission=$(curl --fail --silent --show-error \
+ -H "Authorization: Bearer ${GITHUB_TOKEN}" \
+ "https://api.github.com/repos/${{ github.repository
}}/collaborators/${{ github.event.comment.user.login }}/permission" | jq -r
.permission)
+ [[ "$permission" == write || "$permission" == maintain ||
"$permission" == admin ]] || {
+ echo 'Repository write access is required to trigger current
TeamCity'; exit 1;
+ }
+ fi
+ ;;
+ *) echo "No TeamCity PR pipeline configured for $TARGET_BRANCH";
exit 1;;
+ esac
COMMENT_BODY_DELIMITER=$(dd if=/dev/urandom bs=15 count=1 status=none
| base64)
{
echo "COMMENT_BODY<<${COMMENT_BODY_DELIMITER}"
@@ -181,26 +130,34 @@ jobs:
echo "${COMMENT_BODY_DELIMITER}"
} | tee -a "$GITHUB_OUTPUT"
- reg="run
(buildall|compile|p0|p1|feut|beut|cloudut|external|clickbench|cloud_p0|cloud_p1|vault_p0|nonConcurrent|performance|check_coverage)(
[1-9]*[0-9]+)*"
- COMMENT_TRIGGER_TYPE="$(echo -e "${COMMENT_BODY}" | xargs | grep -E
"${reg}" | awk -F' ' '{print $2}' | sed -n 1p | sed 's/\r//g')"
- COMMENT_REPEAT_TIMES="$(echo -e "${COMMENT_BODY}" | xargs | grep -E
"${reg}" | awk -F' ' '{print $3}' | sed -n 1p | sed 's/\r//g')"
- if [[ -n "${COMMENT_REPEAT_TIMES}" && ! "${COMMENT_REPEAT_TIMES}" =~
^[0-9]+$ ]]; then
- echo "COMMENT_REPEAT_TIMES '${COMMENT_REPEAT_TIMES}' is not a valid
number, ignoring."
- COMMENT_REPEAT_TIMES=""
+ COMMENT_TRIGGER_TYPE=""
+ COMMENT_REPEAT_TIMES="1"
+ reg="^run
(buildall|compile|p0|feut|beut|cloudut|external|cloud_p0|vault_p0|nonconcurrent|performance)(
([1-9][0-9]*))?$"
+ if [[ "${COMMENT_BODY}" =~ $reg ]]; then
+ COMMENT_TRIGGER_TYPE=${BASH_REMATCH[1]}
+ COMMENT_REPEAT_TIMES=${BASH_REMATCH[3]:-1}
+ elif [[ "${COMMENT_BODY}" != 'skip buildall' ]]; then
+ echo 'Unsupported command; use run <TeamCity check suffix> [count]
or run buildall'; exit 1
+ fi
+ if [[ "$COMMENT_TRIGGER_TYPE" == performance ]]; then
+ [[ "$TARGET_BRANCH" == master ]] || { echo 'Performance is available
only for master'; exit 1; }
fi
echo "COMMENT_TRIGGER_TYPE=${COMMENT_TRIGGER_TYPE}" | tee -a
"$GITHUB_OUTPUT"
echo "COMMENT_REPEAT_TIMES=${COMMENT_REPEAT_TIMES}" | tee -a
"$GITHUB_OUTPUT"
- name: "Checkout master"
- if: ${{ fromJSON(steps.parse.outputs.comment_trigger) ||
fromJSON(steps.parse.outputs.comment_skip) ||
fromJSON(steps.parse.outputs.comment_skip_coverage) }}
+ if: ${{ fromJSON(steps.parse.outputs.comment_trigger) ||
fromJSON(steps.parse.outputs.comment_skip) }}
uses: actions/checkout@v4
+ with:
+ ref: ${{ github.event.repository.default_branch }}
+ persist-credentials: false
- name: "Check if pr need run build"
if: ${{ fromJSON(steps.parse.outputs.comment_trigger) }}
id: changes
run: |
source regression-test/pipeline/common/github-utils.sh
- set -x
+ set +x
if _get_pr_changed_files "${{ steps.parse.outputs.PULL_REQUEST_NUM
}}"; then
if file_changed_fe_ut; then
echo "changed_fe_ut=true" | tee -a "$GITHUB_OUTPUT"
@@ -220,16 +177,11 @@ jobs:
if file_changed_regression_p0; then
echo "changed_p0=true" | tee -a "$GITHUB_OUTPUT"
echo "changed_external=true" | tee -a "$GITHUB_OUTPUT"
- echo "changed_nonConcurrent=true" | tee -a "$GITHUB_OUTPUT"
+ echo "changed_nonconcurrent=true" | tee -a "$GITHUB_OUTPUT"
else
echo "changed_p0=false" | tee -a "$GITHUB_OUTPUT"
echo "changed_external=false" | tee -a "$GITHUB_OUTPUT"
- echo "changed_nonConcurrent=false" | tee -a "$GITHUB_OUTPUT"
- fi
- if file_changed_regression_p1; then
- echo "changed_p1=true" | tee -a "$GITHUB_OUTPUT"
- else
- echo "changed_p1=false" | tee -a "$GITHUB_OUTPUT"
+ echo "changed_nonconcurrent=false" | tee -a "$GITHUB_OUTPUT"
fi
if file_changed_performance; then
echo "changed_performance=true" | tee -a "$GITHUB_OUTPUT"
@@ -243,16 +195,6 @@ jobs:
echo "changed_cloud_p0=false" | tee -a "$GITHUB_OUTPUT"
echo "changed_vault_p0=false" | tee -a "$GITHUB_OUTPUT"
fi
- if file_changed_cloud_p1; then
- echo "changed_cloud_p1=true" | tee -a "$GITHUB_OUTPUT"
- else
- echo "changed_cloud_p1=false" | tee -a "$GITHUB_OUTPUT"
- fi
- if file_changed_regression_p0 && file_changed_be_ut &&
file_changed_cloud_p0; then
- echo "need_check_coverage=true" | tee -a "$GITHUB_OUTPUT"
- else
- echo "need_check_coverage=false" | tee -a "$GITHUB_OUTPUT"
- fi
else
echo "INFO: failed to _get_pr_changed_files, default trigger all"
echo "changed_fe_ut=true" | tee -a "$GITHUB_OUTPUT"
@@ -260,13 +202,10 @@ jobs:
echo "changed_cloud_ut=true" | tee -a "$GITHUB_OUTPUT"
echo "changed_p0=true" | tee -a "$GITHUB_OUTPUT"
echo "changed_external=true" | tee -a "$GITHUB_OUTPUT"
- echo "changed_p1=true" | tee -a "$GITHUB_OUTPUT"
echo "changed_performance=true" | tee -a "$GITHUB_OUTPUT"
echo "changed_cloud_p0=true" | tee -a "$GITHUB_OUTPUT"
- echo "changed_cloud_p1=true" | tee -a "$GITHUB_OUTPUT"
echo "changed_vault_p0=true" | tee -a "$GITHUB_OUTPUT"
- echo "changed_nonConcurrent=true" | tee -a "$GITHUB_OUTPUT"
- echo "need_check_coverage=true" | tee -a "$GITHUB_OUTPUT"
+ echo "changed_nonconcurrent=true" | tee -a "$GITHUB_OUTPUT"
fi
# - name: "Setup tmate session"
@@ -286,9 +225,8 @@ jobs:
source regression-test/pipeline/common/github-utils.sh
if [[ ! -f all_files ]]; then
- echo "WARNING: all_files not found (GitHub API call likely failed in
previous step)."
- echo "Skipping sensitive check — previous step already defaulted to
trigger-all."
- exit 0
+ echo "Cannot check pipeline script changes: all_files is missing."
+ exit 1
fi
changed_sensitive=false
@@ -310,18 +248,11 @@ jobs:
exit 1
fi
- - name: "skip check coverage"
- if: ${{ fromJSON(steps.parse.outputs.comment_skip_coverage) }}
- run: |
- source ./regression-test/pipeline/common/teamcity-utils.sh
- set -x
- skip_build "${{ steps.parse.outputs.COMMIT_ID_FROM_TRIGGER }}"
check_coverage
-
- - name: "Trigger or Skip feut"
+ - name: "Trigger feut"
if: ${{ fromJSON(steps.parse.outputs.comment_trigger) &&
contains(fromJSON('["feut", "buildall"]'),
steps.parse.outputs.COMMENT_TRIGGER_TYPE) }}
run: |
source ./regression-test/pipeline/common/teamcity-utils.sh
- set -x
+ set +x
trigger_or_skip_build \
"${{ steps.changes.outputs.changed_fe_ut }}" \
"${{ steps.parse.outputs.PULL_REQUEST_NUM }}" \
@@ -331,11 +262,11 @@ jobs:
"${{ steps.parse.outputs.COMMENT_REPEAT_TIMES }}"
- - name: "Trigger or Skip beut"
+ - name: "Trigger beut"
if: ${{ fromJSON(steps.parse.outputs.comment_trigger) &&
contains(fromJSON('["beut", "buildall"]'),
steps.parse.outputs.COMMENT_TRIGGER_TYPE) }}
run: |
source ./regression-test/pipeline/common/teamcity-utils.sh
- set -x
+ set +x
trigger_or_skip_build \
"${{ steps.changes.outputs.changed_be_ut }}" \
"${{ steps.parse.outputs.PULL_REQUEST_NUM }}" \
@@ -344,11 +275,11 @@ jobs:
"beut" \
"${{ steps.parse.outputs.COMMENT_REPEAT_TIMES }}"
- - name: "Trigger or Skip cloudut"
+ - name: "Trigger cloudut"
if: ${{ fromJSON(steps.parse.outputs.comment_trigger) &&
contains(fromJSON('["cloudut", "buildall"]'),
steps.parse.outputs.COMMENT_TRIGGER_TYPE) }}
run: |
source ./regression-test/pipeline/common/teamcity-utils.sh
- set -x
+ set +x
trigger_or_skip_build \
"${{ steps.changes.outputs.changed_cloud_ut }}" \
"${{ steps.parse.outputs.PULL_REQUEST_NUM }}" \
@@ -357,15 +288,15 @@ jobs:
"cloudut" \
"${{ steps.parse.outputs.COMMENT_REPEAT_TIMES }}"
- - name: "Trigger or Skip compile"
+ - name: "Trigger compile"
if: ${{ fromJSON(steps.parse.outputs.comment_trigger) &&
contains(fromJSON('["compile", "buildall"]'),
steps.parse.outputs.COMMENT_TRIGGER_TYPE) }}
run: |
source ./regression-test/pipeline/common/teamcity-utils.sh
- set -x
- if [[ "${{ steps.changes.outputs.changed_p0 }}" == "true" ]] || [[
"${{ steps.changes.outputs.changed_cloud_p0 }}" == "true" ]]; then
- changed="true"
+ set +x
+ if [[ "${{ steps.changes.outputs.changed_p0 }}" == true || "${{
steps.changes.outputs.changed_cloud_p0 }}" == true ]]; then
+ changed=true
else
- changed="false"
+ changed=false
fi
trigger_or_skip_build \
"${changed}" \
@@ -375,14 +306,14 @@ jobs:
"compile" \
"${{ steps.parse.outputs.COMMENT_REPEAT_TIMES }}"
- - name: "Trigger or Skip p0"
+ - name: "Trigger p0"
if: ${{ fromJSON(steps.parse.outputs.comment_trigger) &&
contains(fromJSON('["p0", "buildall"]'),
steps.parse.outputs.COMMENT_TRIGGER_TYPE) }}
run: |
source ./regression-test/pipeline/common/teamcity-utils.sh
if [[ ${{ steps.parse.outputs.COMMENT_TRIGGER_TYPE }} == "buildall"
]]; then
echo "COMMENT_TRIGGER_TYPE is buildall, trigger compile is enough,
compile will trigger p0" && exit
fi
- set -x
+ set +x
trigger_or_skip_build \
"${{ steps.changes.outputs.changed_p0 }}" \
"${{ steps.parse.outputs.PULL_REQUEST_NUM }}" \
@@ -391,31 +322,14 @@ jobs:
"p0" \
"${{ steps.parse.outputs.COMMENT_REPEAT_TIMES }}"
- - name: "Trigger or Skip p1"
- if: ${{ fromJSON(steps.parse.outputs.comment_trigger) &&
contains(fromJSON('["p1", "buildall"]'),
steps.parse.outputs.COMMENT_TRIGGER_TYPE) }}
- run: |
- source ./regression-test/pipeline/common/teamcity-utils.sh
- COMMIT_ID_FROM_TRIGGER=${{ steps.parse.outputs.COMMIT_ID_FROM_TRIGGER
}}
- if [[ ${{ steps.parse.outputs.COMMENT_TRIGGER_TYPE }} == "buildall"
]]; then
- echo "COMMENT_TRIGGER_TYPE is buildall, trigger compile is enough,
compile will trigger p1" && exit
- fi
- set -x
- trigger_or_skip_build \
- "${{ steps.changes.outputs.changed_p1 }}" \
- "${{ steps.parse.outputs.PULL_REQUEST_NUM }}" \
- "${{ steps.parse.outputs.TARGET_BRANCH }}" \
- "${{ steps.parse.outputs.COMMIT_ID_FROM_TRIGGER }}" \
- "p1" \
- "${{ steps.parse.outputs.COMMENT_REPEAT_TIMES }}"
-
- - name: "Trigger or Skip external"
+ - name: "Trigger external"
if: ${{ fromJSON(steps.parse.outputs.comment_trigger) &&
contains(fromJSON('["external", "buildall"]'),
steps.parse.outputs.COMMENT_TRIGGER_TYPE) }}
run: |
source ./regression-test/pipeline/common/teamcity-utils.sh
if [[ ${{ steps.parse.outputs.COMMENT_TRIGGER_TYPE }} == "buildall"
]]; then
echo "COMMENT_TRIGGER_TYPE is buildall, trigger compile is enough,
compile will trigger external" && exit
fi
- set -x
+ set +x
trigger_or_skip_build \
"${{ steps.changes.outputs.changed_external }}" \
"${{ steps.parse.outputs.PULL_REQUEST_NUM }}" \
@@ -424,14 +338,14 @@ jobs:
"external" \
"${{ steps.parse.outputs.COMMENT_REPEAT_TIMES }}"
- - name: "Trigger or Skip cloud_p0"
+ - name: "Trigger cloud_p0"
if: ${{ fromJSON(steps.parse.outputs.comment_trigger) &&
contains(fromJSON('["cloud_p0", "buildall"]'),
steps.parse.outputs.COMMENT_TRIGGER_TYPE) }}
run: |
source ./regression-test/pipeline/common/teamcity-utils.sh
if [[ ${{ steps.parse.outputs.COMMENT_TRIGGER_TYPE }} == "buildall"
]]; then
echo "COMMENT_TRIGGER_TYPE is buildall, trigger compile is enough,
compile will trigger cloud_p0" && exit
fi
- set -x
+ set +x
trigger_or_skip_build \
"${{ steps.changes.outputs.changed_cloud_p0 }}" \
"${{ steps.parse.outputs.PULL_REQUEST_NUM }}" \
@@ -441,14 +355,14 @@ jobs:
"${{ steps.parse.outputs.COMMENT_REPEAT_TIMES }}"
- - name: "Trigger or Skip vault_p0"
+ - name: "Trigger vault_p0"
if: ${{ fromJSON(steps.parse.outputs.comment_trigger) &&
contains(fromJSON('["vault_p0", "buildall"]'),
steps.parse.outputs.COMMENT_TRIGGER_TYPE) }}
run: |
source ./regression-test/pipeline/common/teamcity-utils.sh
if [[ ${{ steps.parse.outputs.COMMENT_TRIGGER_TYPE }} == "buildall"
]]; then
echo "COMMENT_TRIGGER_TYPE is buildall, trigger compile is enough,
compile will trigger vault_p0" && exit
fi
- set -x
+ set +x
trigger_or_skip_build \
"${{ steps.changes.outputs.changed_vault_p0 }}" \
"${{ steps.parse.outputs.PULL_REQUEST_NUM }}" \
@@ -457,59 +371,27 @@ jobs:
"vault_p0" \
"${{ steps.parse.outputs.COMMENT_REPEAT_TIMES }}"
- - name: "Trigger or Skip nonConcurrent"
- if: ${{ fromJSON(steps.parse.outputs.comment_trigger) &&
contains(fromJSON('["nonConcurrent", "buildall"]'),
steps.parse.outputs.COMMENT_TRIGGER_TYPE) }}
+ - name: "Trigger nonconcurrent"
+ if: ${{ fromJSON(steps.parse.outputs.comment_trigger) &&
contains(fromJSON('["nonconcurrent", "buildall"]'),
steps.parse.outputs.COMMENT_TRIGGER_TYPE) }}
run: |
source ./regression-test/pipeline/common/teamcity-utils.sh
if [[ ${{ steps.parse.outputs.COMMENT_TRIGGER_TYPE }} == "buildall"
]]; then
- echo "COMMENT_TRIGGER_TYPE is buildall, trigger compile is enough,
compile will trigger nonConcurrent" && exit
+ echo "COMMENT_TRIGGER_TYPE is buildall, trigger compile is enough,
compile will trigger nonconcurrent" && exit
fi
- set -x
+ set +x
trigger_or_skip_build \
- "${{ steps.changes.outputs.changed_nonConcurrent }}" \
+ "${{ steps.changes.outputs.changed_nonconcurrent }}" \
"${{ steps.parse.outputs.PULL_REQUEST_NUM }}" \
"${{ steps.parse.outputs.TARGET_BRANCH }}" \
"${{ steps.parse.outputs.COMMIT_ID_FROM_TRIGGER }}" \
- "nonConcurrent" \
+ "nonconcurrent" \
"${{ steps.parse.outputs.COMMENT_REPEAT_TIMES }}"
- - name: "Trigger or Skip check_coverage"
- if: ${{ fromJSON(steps.parse.outputs.comment_trigger) &&
contains(fromJSON('["check_coverage", "buildall"]'),
steps.parse.outputs.COMMENT_TRIGGER_TYPE) }}
+ - name: "Trigger performance"
+ if: ${{ fromJSON(steps.parse.outputs.comment_trigger) &&
contains(fromJSON('["performance", "buildall"]'),
steps.parse.outputs.COMMENT_TRIGGER_TYPE) && steps.parse.outputs.TARGET_BRANCH
== 'master' }}
run: |
source ./regression-test/pipeline/common/teamcity-utils.sh
- if [[ ${{ steps.parse.outputs.COMMENT_TRIGGER_TYPE }} == "buildall"
]]; then
- echo "COMMENT_TRIGGER_TYPE is buildall, check_coverage will be
triggered after p0,cloud_p0,external,nonConcurrent,beut" && exit
- fi
- set -x
- trigger_or_skip_build \
- "${{ steps.changes.outputs.need_check_coverage }}" \
- "${{ steps.parse.outputs.PULL_REQUEST_NUM }}" \
- "${{ steps.parse.outputs.TARGET_BRANCH }}" \
- "${{ steps.parse.outputs.COMMIT_ID_FROM_TRIGGER }}" \
- "check_coverage" \
- "${{ steps.parse.outputs.COMMENT_REPEAT_TIMES }}"
-
- - name: "Trigger or Skip cloud_p1"
- if: ${{ fromJSON(steps.parse.outputs.comment_trigger) &&
contains(fromJSON('["cloud_p1", "buildall"]'),
steps.parse.outputs.COMMENT_TRIGGER_TYPE) }}
- run: |
- source ./regression-test/pipeline/common/teamcity-utils.sh
- if [[ ${{ steps.parse.outputs.COMMENT_TRIGGER_TYPE }} == "buildall"
]]; then
- echo "COMMENT_TRIGGER_TYPE is buildall, trigger compile is enough,
compile will trigger cloud_p1" && exit
- fi
- set -x
- trigger_or_skip_build \
- "${{ steps.changes.outputs.changed_cloud_p1 }}" \
- "${{ steps.parse.outputs.PULL_REQUEST_NUM }}" \
- "${{ steps.parse.outputs.TARGET_BRANCH }}" \
- "${{ steps.parse.outputs.COMMIT_ID_FROM_TRIGGER }}" \
- "cloud_p1" \
- "${{ steps.parse.outputs.COMMENT_REPEAT_TIMES }}"
-
- - name: "Trigger or Skip performance"
- if: ${{ fromJSON(steps.parse.outputs.comment_trigger) &&
contains(fromJSON('["performance", "buildall"]'),
steps.parse.outputs.COMMENT_TRIGGER_TYPE) }}
- run: |
- source ./regression-test/pipeline/common/teamcity-utils.sh
- set -x
+ set +x
trigger_or_skip_build \
"${{ steps.changes.outputs.changed_performance }}" \
"${{ steps.parse.outputs.PULL_REQUEST_NUM }}" \
@@ -526,12 +408,11 @@ jobs:
skip_build "${{ steps.parse.outputs.COMMIT_ID_FROM_TRIGGER }}" beut
skip_build "${{ steps.parse.outputs.COMMIT_ID_FROM_TRIGGER }}" compile
skip_build "${{ steps.parse.outputs.COMMIT_ID_FROM_TRIGGER }}" p0
- skip_build "${{ steps.parse.outputs.COMMIT_ID_FROM_TRIGGER }}" p1
skip_build "${{ steps.parse.outputs.COMMIT_ID_FROM_TRIGGER }}" external
- skip_build "${{ steps.parse.outputs.COMMIT_ID_FROM_TRIGGER }}"
performance
+ if [[ "$TARGET_BRANCH" == master ]]; then
+ skip_build "${{ steps.parse.outputs.COMMIT_ID_FROM_TRIGGER }}"
performance
+ fi
skip_build "${{ steps.parse.outputs.COMMIT_ID_FROM_TRIGGER }}" cloud_p0
- skip_build "${{ steps.parse.outputs.COMMIT_ID_FROM_TRIGGER }}" cloud_p1
skip_build "${{ steps.parse.outputs.COMMIT_ID_FROM_TRIGGER }}" cloudut
skip_build "${{ steps.parse.outputs.COMMIT_ID_FROM_TRIGGER }}" vault_p0
- skip_build "${{ steps.parse.outputs.COMMIT_ID_FROM_TRIGGER }}"
nonConcurrent
- skip_build "${{ steps.parse.outputs.COMMIT_ID_FROM_TRIGGER }}"
check_coverage
+ skip_build "${{ steps.parse.outputs.COMMIT_ID_FROM_TRIGGER }}"
nonconcurrent
diff --git a/regression-test/pipeline/common/github-utils.sh
b/regression-test/pipeline/common/github-utils.sh
index c5d00a47acd..1dbc8040cc9 100644
--- a/regression-test/pipeline/common/github-utils.sh
+++ b/regression-test/pipeline/common/github-utils.sh
@@ -129,7 +129,7 @@ _get_pr_changed_files_count() {
REPO="${REPO:=doris}"
try_times=10
while [[ ${try_times} -gt 0 ]]; do
- set -x
+ set +x
if ret=$(
curl -s \
-H "Accept: application/vnd.github+json" \
@@ -168,7 +168,7 @@ _get_pr_changed_files() {
while [[ ${changed_files_count} -gt 0 ]]; do
try_times=10
while [[ ${try_times} -gt 0 ]]; do
- set -x
+ set +x
if curl -s \
-H "Accept: application/vnd.github+json" \
${GITHUB_TOKEN:+-H "Authorization: Bearer ${GITHUB_TOKEN}"} \
diff --git a/regression-test/pipeline/common/teamcity-utils.sh
b/regression-test/pipeline/common/teamcity-utils.sh
index 90c8749527a..df8d84ffb83 100644
--- a/regression-test/pipeline/common/teamcity-utils.sh
+++ b/regression-test/pipeline/common/teamcity-utils.sh
@@ -16,252 +16,94 @@
# specific language governing permissions and limitations
# under the License.
-#!/bin/bash
-
-# PR的目标分支
-# 到
-# 可以跑的流水线
-# 的映射
-# 控制哪些分支可以跑哪些流水线
-declare -A targetBranch_to_pipelines
-targetBranch_to_pipelines=(
- ['master']='feut beut cloudut compile p0 p1 external performance cloud_p0
cloud_p1 vault_p0 nonConcurrent check_coverage check_coverage_fe'
- ['branch-4.2']='feut beut cloudut compile p0 p1 external cloud_p0 cloud_p1
vault_p0 nonConcurrent check_coverage check_coverage_fe'
- ['branch-4.1']='feut beut cloudut compile p0 p1 external cloud_p0 cloud_p1
vault_p0 nonConcurrent check_coverage check_coverage_fe'
- ['branch-4.0']='feut beut cloudut compile p0 p1 external cloud_p0 cloud_p1
vault_p0 nonConcurrent check_coverage check_coverage_fe'
- ['branch-3.1']='feut beut cloudut compile p0 p1 external cloud_p0 cloud_p1
vault_p0 nonConcurrent check_coverage check_coverage_fe'
- ['branch-3.0']='feut beut cloudut compile p0 p1 external cloud_p0 cloud_p1
vault_p0 nonConcurrent check_coverage'
- ['branch-2.1']='feut beut compile p0 p1 external'
- ['branch-2.0']='feut beut compile p0 p1 external'
+# Comment keywords match the suffix of the published TeamCity/<keyword> check.
+# External and Performance are optional and therefore absent from required
checks.
+declare -A targetBranch_to_pipelines=(
+ [master]='compile feut beut cloudut p0 cloud_p0 vault_p0 external
nonconcurrent performance'
+ [branch-4.2]='compile feut beut cloudut p0 cloud_p0 vault_p0 external
nonconcurrent'
+ [branch-4.1]='compile feut beut cloudut p0 cloud_p0 vault_p0 external
nonconcurrent'
)
-
-# github中评论的要触发的流水线名字
-# 到
-# teamcity流水线实际的名称
-# 的映射
-# 新加流水线需要修改这里
-declare -A comment_to_pipeline
-comment_to_pipeline=(
- ['feut']='Doris_Doris_FeUt'
- ['beut']='Doris_DorisBeUt_BeUt'
- ['cloudut']='Doris_DorisCloudUt_CloudUt'
- ['compile']='Doris_DorisCompile_Compile'
- ['p0']='Doris_DorisRegression_P0Regression'
- ['p1']='Doris_DorisRegression_P1Regression'
- ['external']='Doris_External_Regression'
- ['performance']='Doris_DorisPerformance_Performance'
- ['cloud_p0']='Doris_DorisRegression_CloudP0'
- ['cloud_p1']='Doris_DorisCloudRegression_CloudP1'
- ['vault_p0']='Doris_DorisCloudRegression_VaultP0'
- ['nonConcurrent']='Doris_DorisRegression_NonConcurrentRegression'
- ['check_coverage']='Doris_Coverage_Merge_P0_UT'
+declare -A comment_to_pipeline=(
+ [compile]=Doris_Compile [feut]=Doris_FEUT [beut]=Doris_BEUT
+ [cloudut]=Doris_CloudUT [p0]=Doris_P0 [cloud_p0]=Doris_CloudP0
+ [vault_p0]=Doris_VaultP0 [external]=Doris_External
+ [nonconcurrent]=Doris_NonConcurrent [performance]=Doris_Performance
)
-
-# github中评论的要触发的流水线名字
-# 到
-# teamcity流水线返回结果给github的名称
-# 的映射
-# 新加流水线需要修改这里
declare -A conment_to_context
-conment_to_context=(
- ['compile']='COMPILE (DORIS_COMPILE)'
- ['feut']='FE UT (Doris FE UT)'
- ['beut']='BE UT (Doris BE UT)'
- ['cloudut']='Cloud UT (Doris Cloud UT)'
- ['p0']='P0 Regression (Doris Regression)'
- ['p1']='P1 Regression (Doris Regression)'
- ['external']='External Regression (Doris External Regression)'
- ['performance']='performance (Doris Performance)'
- ['cloud_p0']='cloud_p0 (Doris Cloud Regression)'
- ['cloud_p1']='cloud_p1 (Doris Cloud Regression)'
- ['vault_p0']='vault_p0 (Doris Cloud Regression)'
- ['nonConcurrent']='NonConcurrent Regression (nonConcurrent)'
- ['check_coverage']='check_coverage (Coverage)'
-)
-
-get_commit_id_of_build() {
- # 获取某个build的commit id
- if [[ -z "$1" ]]; then return 1; fi
- local build_id="$1"
- local commit_id
- local ret
- set -x
- if ret=$(
- curl -s -X GET \
- -u OneMoreChance:OneMoreChance \
- -H "Content-Type:text/plain" \
- -H "Accept: application/json" \
- "http://43.132.222.7:8111/app/rest/builds/${build_id}";
- ); then
- set +x
- commit_id=$(echo "${ret}" | jq -r '.revisions.revision[0].version')
- echo "${commit_id}"
- else
- set +x
- return 1
- fi
+for lane in "${!comment_to_pipeline[@]}"; do
conment_to_context[$lane]="TeamCity/$lane"; done
+unset lane
+TEAMCITY_URL=${TEAMCITY_CURRENT_URL:-}
+
+# Never trace authenticated requests or follow redirects carrying credentials.
+teamcity_request() {
+ set +x
+ [[ ${TEAMCITY_URL:-} =~ ^https://[A-Za-z0-9.-]+(:[0-9]+)?$ && -n
${TEAMCITY_CURRENT_TOKEN:-} ]] || {
+ echo 'TeamCity HTTPS URL/token is not configured' >&2; return 1;
+ }
+ curl --header "Authorization: Bearer ${TEAMCITY_CURRENT_TOKEN}" "$@"
}
-get_running_build_of_pr() {
- # "获取pr在某条流水线上正在跑的build"
- local PULL_REQUEST_NUM="${PULL_REQUEST_NUM:-$1}"
- local COMMENT_TRIGGER_TYPE="${COMMENT_TRIGGER_TYPE:-$2}"
- if [[ -z "${PULL_REQUEST_NUM}" || -z "${COMMENT_TRIGGER_TYPE}" ]]; then
- echo "Usage: get_queue_build_of_pr PULL_REQUEST_NUM
COMMENT_TRIGGER_TYPE" && return 1
- fi
-
- local PIPELINE="${comment_to_pipeline[${COMMENT_TRIGGER_TYPE}]}"
- local running_builds_list
- local ret
- set -x
- if ret=$(
- curl -s -X GET \
- -u OneMoreChance:OneMoreChance \
- -H "Content-Type:text/plain" \
- -H "Accept: application/json" \
-
"http://43.132.222.7:8111/app/rest/builds?locator=buildType:${PIPELINE},branch:pull/${PULL_REQUEST_NUM},running:true";
- ); then
- set +x
- running_builds_list=$(echo "${ret}" | jq -r '.build[].id')
- echo "${running_builds_list}"
- else
- set +x
- return 1
- fi
+teamcity_rest() {
+ local path=$1; shift
+ teamcity_request --globoff --fail --silent --show-error --max-time 45
--max-filesize 8388608 \
+ -H 'Accept: application/json' -H 'Content-Type: application/json' \
+ "${TEAMCITY_URL}/app/rest/${path}" "$@"
}
-# get_running_build_of_pr "$1" "$2"
-
-get_queue_build_of_pr() {
- # "获取pr在某条流水线上排队的build"
- local PULL_REQUEST_NUM="${PULL_REQUEST_NUM:-$1}"
- local COMMENT_TRIGGER_TYPE="${COMMENT_TRIGGER_TYPE:-$2}"
- if [[ -z "${PULL_REQUEST_NUM}" || -z "${COMMENT_TRIGGER_TYPE}" ]]; then
- echo "Usage: get_queue_build_of_pr PULL_REQUEST_NUM
COMMENT_TRIGGER_TYPE" && return 1
- fi
- local PIPELINE="${comment_to_pipeline[${COMMENT_TRIGGER_TYPE}]}"
- local queue_builds_list
- local ret
- set -x
- if ret=$(
- curl -s -X GET \
- -u OneMoreChance:OneMoreChance \
- -H "Content-Type:text/plain" \
- -H "Accept: application/json" \
-
"http://43.132.222.7:8111/app/rest/buildQueue?locator=buildType:${PIPELINE}";
- ); then
- set +x
- queue_builds_list=$(echo "${ret}" | jq ".build[] | select(.branchName
== \"pull/${PULL_REQUEST_NUM}\") | .id")
- echo "${queue_builds_list}"
- else
- set +x
- echo "WARNING: failed to get queue build for PR ${PULL_REQUEST_NUM} of
pipeline ${PIPELINE}" && return 1
- fi
+teamcity_current_pr() {
+ set +x
+ curl --fail --silent --show-error --max-time 45 \
+ -H "Authorization: Bearer ${GITHUB_TOKEN:?}" \
+ "https://api.github.com/repos/apache/doris/pulls/${PULL_REQUEST_NUM}"; |
+ jq -e --arg head "$COMMIT_ID_FROM_TRIGGER" --arg branch
"$PULL_REQUEST_TARGET_BRANCH" \
+ '.state=="open" and .draft==false and .head.sha==$head and
.base.ref==$branch' >/dev/null
}
-# get_queue_build_of_pr "$1" "$2"
-
-get_active_builds_of_revision() {
- # Return active build IDs for the same PR, pipeline, and revision.
- # Return 0 when a duplicate exists, 1 when none exists, and 2 when lookup
fails.
- local PULL_REQUEST_NUM="${PULL_REQUEST_NUM:-$1}"
- local COMMENT_TRIGGER_TYPE="${COMMENT_TRIGGER_TYPE:-$2}"
- local COMMIT_ID_FROM_TRIGGER="${COMMIT_ID_FROM_TRIGGER:-$3}"
- if [[ -z "${PULL_REQUEST_NUM}" ||
- -z "${COMMENT_TRIGGER_TYPE}" ||
- -z "${COMMIT_ID_FROM_TRIGGER}" ]]; then
- echo "Usage: get_active_builds_of_revision PULL_REQUEST_NUM
COMMENT_TRIGGER_TYPE COMMIT_ID_FROM_TRIGGER" >&2
- return 2
- fi
-
- local queue_build_ids
- local running_build_ids
- if ! queue_build_ids=$(get_queue_build_of_pr "${PULL_REQUEST_NUM}"
"${COMMENT_TRIGGER_TYPE}"); then
- echo "WARNING: failed to get queued builds for duplicate check" >&2
- return 2
- fi
- if ! running_build_ids=$(get_running_build_of_pr "${PULL_REQUEST_NUM}"
"${COMMENT_TRIGGER_TYPE}"); then
- echo "WARNING: failed to get running builds for duplicate check" >&2
- return 2
- fi
-
- local build_id
- local build_revision
- local duplicate_build_ids=()
- for build_id in ${queue_build_ids} ${running_build_ids}; do
- if ! build_revision=$(get_commit_id_of_build "${build_id}"); then
- echo "WARNING: failed to get revision of active build ${build_id}"
>&2
- return 2
- fi
- if [[ "${build_revision}" == "${COMMIT_ID_FROM_TRIGGER}" ]]; then
- duplicate_build_ids+=("${build_id}")
- fi
- done
- if [[ ${#duplicate_build_ids[@]} -gt 0 ]]; then
- printf '%s\n' "${duplicate_build_ids[@]}"
- return 0
- fi
- return 1
+teamcity_current_root() {
+ teamcity_rest "buildTypes/id:$1/vcs-root-entries" |
+ jq -e '."vcs-root-entry" | length==1 and
.[0]."vcs-root".id=="DorisMigrationValidation_ApacheDoris"' >/dev/null || {
+ echo "$1 is not configured for general PR builds; no build
submitted" >&2; return 1;
+ }
}
-# get_active_builds_of_revision "$1" "$2" "$3"
-
-cancel_running_build() {
- local PULL_REQUEST_NUM="${PULL_REQUEST_NUM:-$1}"
- local COMMENT_TRIGGER_TYPE="${COMMENT_TRIGGER_TYPE:-$2}"
- if [[ -z "${PULL_REQUEST_NUM}" || -z "${COMMENT_TRIGGER_TYPE}" ]]; then
- echo "Usage: get_queue_build_of_pr PULL_REQUEST_NUM
COMMENT_TRIGGER_TYPE" && return 1
- fi
- local PIPELINE="${comment_to_pipeline[${COMMENT_TRIGGER_TYPE}]}"
- local build_ids
- if ! build_ids=$(get_running_build_of_pr "${PULL_REQUEST_NUM}"
"${COMMENT_TRIGGER_TYPE}"); then return 1; fi
- for id in ${build_ids}; do
- set -x
- if curl -s -X POST \
- -u OneMoreChance:OneMoreChance \
- -H "Content-Type:application/json" \
- -H "Accept: application/json" \
- "http://43.132.222.7:8111/app/rest/builds/id:${id}"; \
- -d '{ "comment": "Canceling this running build before triggering a
new one", "readdIntoQueue": false }'; then
- set +x
- echo -e "\nINFO: canceled running build(id ${id}) for PR
${PULL_REQUEST_NUM} of pipeline ${PIPELINE}"
- else
- set +x
- echo "WARNING: failed to cancel running build(id ${id}) for PR
${PULL_REQUEST_NUM} of pipeline ${PIPELINE}"
- fi
+teamcity_current_enqueue() {
+ local type=$1 params=$2 count=$3 ordinal id instance payload actual
+ teamcity_current_root "$type" || return 1
+ instance=$(teamcity_rest
"vcs-root-instances?locator=buildType:(id:${type})&fields=vcs-root-instance(id,vcs-root-id)"
|
+ jq -er '[."vcs-root-instance"[] |
select(."vcs-root-id"=="DorisMigrationValidation_ApacheDoris")] |
select(length==1) | .[0].id') || return 1
+ for ((ordinal=1; ordinal<=count; ordinal++)); do
+ teamcity_current_pr || { echo 'PR revision or target changed; no new
build submitted' >&2; return 1; }
+ payload=$(jq -cn --arg type "$type" --arg pr "$PULL_REQUEST_NUM" --arg
head "$COMMIT_ID_FROM_TRIGGER" --arg instance "$instance" --argjson params
"$params" '
+
{buildType:{id:$type},branchName:("pull/"+$pr),revisions:{failOnMissingRevisions:true,revision:[{version:$head,vcsBranchName:("refs/pull/"+$pr+"/head"),"vcs-root-instance":{id:$instance}}]},
+
properties:{property:($params|to_entries|map({name:.key,value:.value}))},
+
triggeringOptions:{queueAtTop:false,rebuildAllDependencies:false}}') || return 1
+ # Queue replacement and duplicate handling belong to TeamCity.
+ id=$(teamcity_rest buildQueue -X POST --data-binary "$payload" | jq
-er .id) || return 1
+ actual=$(teamcity_rest "builds/id:${id}?fields=${TC_FIELDS}") ||
return 1
+ jq -e --arg head "$COMMIT_ID_FROM_TRIGGER" --argjson params "$params" '
+ ((.properties.property // []) | map({key:.name,value:.value}) |
from_entries) as $p |
+ [.revisions.revision[].version]==[$head] and
all($params|to_entries[]; $p[.key]==.value)' <<< "$actual" >/dev/null || return
1
+ echo "INFO: ${type} build ${id}:
${TEAMCITY_URL}/buildConfiguration/${type}/${id}" >&2
+ printf '%s\n' "$id"
done
}
-# cancel_running_build "$1" "$2"
-cancel_queue_build() {
- local PULL_REQUEST_NUM="${PULL_REQUEST_NUM:-$1}"
- local COMMENT_TRIGGER_TYPE="${COMMENT_TRIGGER_TYPE:-$2}"
- if [[ -z "${PULL_REQUEST_NUM}" || -z "${COMMENT_TRIGGER_TYPE}" ]]; then
- echo "Usage: get_queue_build_of_pr PULL_REQUEST_NUM
COMMENT_TRIGGER_TYPE" && return 1
- fi
-
- local PIPELINE="${comment_to_pipeline[${COMMENT_TRIGGER_TYPE}]}"
- local build_ids
- if ! build_ids=$(get_queue_build_of_pr "${PULL_REQUEST_NUM}"
"${COMMENT_TRIGGER_TYPE}"); then return 1; fi
- for id in ${build_ids}; do
- set -x
- if curl -s -X POST \
- -u OneMoreChance:OneMoreChance \
- -H "Content-Type:application/json" \
- -H "Accept: application/json" \
- "http://43.132.222.7:8111/app/rest/buildQueue/id:${id}"; \
- -d '{ "comment": "Canceling this queued build before triggering a
new one", "readdIntoQueue": false }'; then
- set +x
- echo -e "\nINFO: canceled queue build(id ${id}) for PR
${PULL_REQUEST_NUM} of pipeline ${PIPELINE}"
- else
- set +x
- echo "WARNING: failed to cancel queue build(id ${id}) for PR
${PULL_REQUEST_NUM} of pipeline ${PIPELINE}"
- fi
- done
-}
-# cancel_queue_build "$1" "$2"
+teamcity_current_trigger() (
+ set -euo pipefail
+ set +x
+ local lane=$1 count=${2:-1} params
+ [[ $PULL_REQUEST_NUM =~ ^[1-9][0-9]*$ && $COMMIT_ID_FROM_TRIGGER =~
^[0-9a-f]{40}$ && $count =~ ^[1-9][0-9]*$ && $count -le 20 ]]
+ [[ " ${targetBranch_to_pipelines[$PULL_REQUEST_TARGET_BRANCH]:-} " == *"
$lane "* ]] || {
+ echo "No pipeline for $lane on $PULL_REQUEST_TARGET_BRANCH" >&2; exit
1;
+ }
+ # Artifact lookup belongs to the pipeline; only submit the requested lane
here.
+ local
TC_FIELDS='id,revisions(revision(version)),properties(property(name,value))'
+ params=$(jq -cn --arg pr "$PULL_REQUEST_NUM" --arg branch
"$PULL_REQUEST_TARGET_BRANCH" '{"env.PR_NUMBER":$pr,"env.CI_BRANCH":$branch}')
+ teamcity_current_enqueue "${comment_to_pipeline[$lane]}" "$params" "$count"
+)
skip_build() {
+ set +x
# 对于不需要跑teamcity pipeline的PR，直接调用github的接口返回成功
if [[ -z "${GITHUB_TOKEN}" ]]; then
echo "ERROR: env GITHUB_TOKEN not set"
@@ -274,10 +116,14 @@ skip_build() {
local COMMIT_ID_FROM_TRIGGER="$1"
local COMMENT_TRIGGER_TYPE="$2"
+ if [[ -z ${conment_to_context[$COMMENT_TRIGGER_TYPE]:-} ]]; then
+ echo "ERROR: Unknown TeamCity check ${COMMENT_TRIGGER_TYPE}" >&2
+ return 1
+ fi
local state="${TC_BUILD_STATE:-success}" # 可选值 success failure pending
local
payload="{\"state\":\"${state}\",\"target_url\":\"\",\"description\":\"Skip
teamCity
build\",\"context\":\"${conment_to_context[${COMMENT_TRIGGER_TYPE}]}\"}"
- set -x
- if curl -L \
+ set +x
+ if curl --fail --silent --show-error --max-time 45 \
-X POST \
-H "Accept: application/vnd.github+json" \
-H "Authorization: Bearer ${GITHUB_TOKEN:-}" \
@@ -299,29 +145,31 @@ trigger_build() {
local COMMIT_ID_FROM_TRIGGER="${COMMIT_ID_FROM_TRIGGER:-$2}"
local COMMENT_TRIGGER_TYPE="${COMMENT_TRIGGER_TYPE:-$3}"
local COMMENT_REPEAT_TIMES="${COMMENT_REPEAT_TIMES:-$4}"
+ local
PULL_REQUEST_TARGET_BRANCH="${PULL_REQUEST_TARGET_BRANCH:-${TARGET_BRANCH:-}}"
if [[ -z "${PULL_REQUEST_NUM}" || -z "${COMMIT_ID_FROM_TRIGGER}" || -z
"${COMMENT_TRIGGER_TYPE}" ]]; then
echo "Usage: add_build PULL_REQUEST_NUM COMMIT_ID_FROM_TRIGGER
COMMENT_TRIGGER_TYPE [COMMENT_REPEAT_TIMES]"
return 1
fi
- local PIPELINE="${comment_to_pipeline[${COMMENT_TRIGGER_TYPE}]}"
- set -x
- if curl -s -X POST \
- -u OneMoreChance:OneMoreChance \
- -H "Content-Type:text/plain" \
- -H "Accept: application/json" \
-
"http://43.132.222.7:8111/httpAuth/action.html?add2Queue=${PIPELINE}&branchName=pull/${PULL_REQUEST_NUM}&name=env.pr_num_from_trigger&value=${PULL_REQUEST_NUM:-}&name=env.commit_id_from_trigger&value=${COMMIT_ID_FROM_TRIGGER:-}&name=env.repeat_times_from_trigger&value=${COMMENT_REPEAT_TIMES:-1}";;
then
- set +x
- echo "INFO: Add new build to PIPELINE ${PIPELINE} of PR
${PULL_REQUEST_NUM} with COMMENT_REPEAT_TIMES ${COMMENT_REPEAT_TIMES:-1}"
- else
- set +x
- return 1
- fi
+ teamcity_current_trigger "$COMMENT_TRIGGER_TYPE"
"${COMMENT_REPEAT_TIMES:-1}"
}
# trigger_build "$1" "$2" "$3" "$4"
+# A skipped Compile also resolves the checks that its fanout would publish.
+teamcity_skip_unaffected() {
+ teamcity_current_pr || { echo 'PR revision or target changed; no skip
status written' >&2; return 1; }
+ local lane lanes="$COMMENT_TRIGGER_TYPE"
+ if [[ $COMMENT_TRIGGER_TYPE == compile ]]; then
+ lanes='compile p0 cloud_p0 vault_p0 external nonconcurrent'
+ fi
+ for lane in $lanes; do
+ skip_build "$COMMIT_ID_FROM_TRIGGER" "$lane" || return 1
+ done
+}
+
trigger_or_skip_build() {
- # 根据相关文件是否修改，来触发or跳过跑流水线
- local FILE_CHANGED="$1" # 默认为"true"
+ set +x
+ # Preserve the existing per-pipeline file filter.
+ local FILE_CHANGED="${1:-true}"
local PULL_REQUEST_NUM="${PULL_REQUEST_NUM:-$2}"
local PULL_REQUEST_TARGET_BRANCH="${PULL_REQUEST_TARGET_BRANCH:-$3}"
local COMMIT_ID_FROM_TRIGGER="${COMMIT_ID_FROM_TRIGGER:-$4}"
@@ -334,55 +182,14 @@ trigger_or_skip_build() {
return 1
fi
- # 有些分支不需要某些跑流水线，在targetBranch_to_pipelines中的分支，需要判断
- if [[ "${!targetBranch_to_pipelines[*]}" =~ ${PULL_REQUEST_TARGET_BRANCH}
]]; then
- if [[ "${targetBranch_to_pipelines[${PULL_REQUEST_TARGET_BRANCH}]}" =~
${COMMENT_TRIGGER_TYPE} ]]; then
- echo "INFO: pr ${PULL_REQUEST_NUM} to branch
${PULL_REQUEST_TARGET_BRANCH} need to run ${COMMENT_TRIGGER_TYPE}"
- else
- echo "INFO: pr ${PULL_REQUEST_NUM} to branch
${PULL_REQUEST_TARGET_BRANCH} not need to run ${COMMENT_TRIGGER_TYPE}"
- skip_build "${COMMIT_ID_FROM_TRIGGER}" "${COMMENT_TRIGGER_TYPE}"
- return 0
- fi
- fi
-
- if [[ "${FILE_CHANGED:-"true"}" == "true" ]]; then
- local duplicate_build_ids
- local duplicate_lookup_status=0
- duplicate_build_ids=$(
- get_active_builds_of_revision \
- "${PULL_REQUEST_NUM}" \
- "${COMMENT_TRIGGER_TYPE}" \
- "${COMMIT_ID_FROM_TRIGGER}"
- ) || duplicate_lookup_status=$?
- if [[ ${duplicate_lookup_status} -eq 0 ]]; then
- echo "INFO: active build(s) ${duplicate_build_ids//$'\n'/,}
already exist for PR ${PULL_REQUEST_NUM}, pipeline ${COMMENT_TRIGGER_TYPE},
revision ${COMMIT_ID_FROM_TRIGGER}; skip duplicate trigger"
- return 0
- elif [[ ${duplicate_lookup_status} -ne 1 ]]; then
- echo "WARNING: duplicate lookup failed for PR ${PULL_REQUEST_NUM},
pipeline ${COMMENT_TRIGGER_TYPE}; continue with the existing trigger flow"
- fi
-
- cancel_running_build "${PULL_REQUEST_NUM}" "${COMMENT_TRIGGER_TYPE}"
- cancel_queue_build "${PULL_REQUEST_NUM}" "${COMMENT_TRIGGER_TYPE}"
- trigger_build "${PULL_REQUEST_NUM}" "${COMMIT_ID_FROM_TRIGGER}"
"${COMMENT_TRIGGER_TYPE}" "${COMMENT_REPEAT_TIMES}"
- else
- cancel_running_build "${PULL_REQUEST_NUM}" "${COMMENT_TRIGGER_TYPE}"
- cancel_queue_build "${PULL_REQUEST_NUM}" "${COMMENT_TRIGGER_TYPE}"
- skip_build "${COMMIT_ID_FROM_TRIGGER}" "${COMMENT_TRIGGER_TYPE}"
- if [[ ${COMMENT_TRIGGER_TYPE} == "compile" ]]; then
- # skip compile 的时候，也把 p0 p1 external cloud_p0 cloud_p1 vault_p0
nonConcurrent check_coverage 都 skip 了
- skip_build "${COMMIT_ID_FROM_TRIGGER}" "p0"
- skip_build "${COMMIT_ID_FROM_TRIGGER}" "p1"
- skip_build "${COMMIT_ID_FROM_TRIGGER}" "external"
- skip_build "${COMMIT_ID_FROM_TRIGGER}" "cloud_p0"
- skip_build "${COMMIT_ID_FROM_TRIGGER}" "cloud_p1"
- skip_build "${COMMIT_ID_FROM_TRIGGER}" "vault_p0"
- skip_build "${COMMIT_ID_FROM_TRIGGER}" "nonConcurrent"
- skip_build "${COMMIT_ID_FROM_TRIGGER}" "check_coverage"
- elif [[ ${COMMENT_TRIGGER_TYPE} == "beut" ]]; then
- # skip beut 的时候，也把 check_coverage skip 了
- skip_build "${COMMIT_ID_FROM_TRIGGER}" "check_coverage"
- fi
- fi
+ [[ " ${targetBranch_to_pipelines[$PULL_REQUEST_TARGET_BRANCH]:-} " == *"
$COMMENT_TRIGGER_TYPE "* ]] || {
+ echo "No pipeline for $COMMENT_TRIGGER_TYPE on
$PULL_REQUEST_TARGET_BRANCH" >&2; return 1;
+ }
+ case "$FILE_CHANGED" in
+ true) trigger_build "$PULL_REQUEST_NUM" "$COMMIT_ID_FROM_TRIGGER"
"$COMMENT_TRIGGER_TYPE" "${COMMENT_REPEAT_TIMES:-1}" ;;
+ false) teamcity_skip_unaffected ;;
+ *) echo "Invalid FILE_CHANGED: $FILE_CHANGED" >&2; return 1 ;;
+ esac
}
# trigger_or_skip_build "$1" "$2" "$3" "$4" "$5"
diff --git a/regression-test/pipeline/common/tests/test_teamcity_utils.py
b/regression-test/pipeline/common/tests/test_teamcity_utils.py
new file mode 100644
index 00000000000..df740ef82cb
--- /dev/null
+++ b/regression-test/pipeline/common/tests/test_teamcity_utils.py
@@ -0,0 +1,352 @@
+#!/usr/bin/env python3
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""Exercise the existing shell entry points without a network or real
credentials."""
+import json
+import os
+import re
+from pathlib import Path
+import subprocess
+import tempfile
+import textwrap
+import unittest
+
+HELPER = Path(__file__).parents[1] / 'teamcity-utils.sh'
+HEAD = 'a' * 40
+MOCK = r'''
+import json, os, re, sys
+from pathlib import Path
+root=Path(os.environ['FIXTURE_ROOT']); path=root/'state.json';
state=json.loads(path.read_text())
+args=sys.argv[1:]; url=next(x for x in args if
x.startswith('https://ci.example/app/rest/'))
+route=url.split('/app/rest/',1)[1]; method=args[args.index('-X')+1] if '-X' in
args else 'GET'
+rows=state.get('builds',[])
+def emit(value): print(json.dumps(value))
+if method=='POST':
+ assert route=='buildQueue'
+ body=json.loads(args[args.index('--data-binary')+1]);
state.setdefault('posts',[]).append(body)
+ bid=100+len(state['posts']);
body.update(id=bid,number=str(bid),buildTypeId=body['buildType']['id'],state='queued',status='UNKNOWN')
+
body['revisions']['revision']=[{'version':body['revisions']['revision'][0]['version']}]
+ state.setdefault('builds',[]).append(body); path.write_text(json.dumps(state))
+ if state.get('lose_response'):
+ state['lose_response']=False; path.write_text(json.dumps(state));sys.exit(1)
+ emit({'id':bid});sys.exit()
+if 'vcs-root-entries' in route:
+
emit({'vcs-root-entry':[{'vcs-root':{'id':state.get('root','DorisMigrationValidation_ApacheDoris')}}]})
+elif route.startswith('vcs-root-instances'):
+
emit({'vcs-root-instance':[{'id':3,'vcs-root-id':'DorisMigrationValidation_ApacheDoris'}]})
+elif route.startswith('builds/id:'):
+ bid=int(route.split(':')[1].split('?')[0]);emit(next(b for b in rows if
b['id']==bid))
+else: raise AssertionError(route)
+'''
+
+
+def build(bid=1, lane='Doris_CloudUT', state='running', head=HEAD,
params=None):
+ return {'id': bid, 'number': str(bid), 'buildTypeId': lane, 'state': state,
+ 'status': 'SUCCESS', 'branchName': 'pull/123',
+ 'revisions': {'revision': [{'version': head}]},
+ 'properties': {'property': [{'name': k, 'value': v} for k, v in
(params or {}).items()]}}
+
+
+class TeamCityUtilsTests(unittest.TestCase):
+ def setUp(self):
+ self.temp = tempfile.TemporaryDirectory()
+ self.addCleanup(self.temp.cleanup)
+ self.root = Path(self.temp.name)
+ (self.root / 'mock.py').write_text(MOCK)
+ (self.root / 'status_mock.py').write_text('''import json, os, sys
+from pathlib import Path
+p=Path(os.environ['FIXTURE_ROOT'])/'statuses.json'
+rows=json.loads(p.read_text()) if p.exists() else []
+assert '-d' in sys.argv and '-X' in sys.argv
+if os.environ.get('STATUS_FAIL')=='true': sys.exit(22)
+rows.append(json.loads(sys.argv[sys.argv.index('-d')+1]));p.write_text(json.dumps(rows))
+''')
+ self.save({})
+ (self.root / 'event.json').write_text(json.dumps({'comment': {
+ 'id': 45, 'body': 'run cloudut', 'updated_at':
'2026-11-11T00:00:00Z'}}))
+ self.env = {**os.environ, 'TARGET_BRANCH': 'master',
'TEAMCITY_CURRENT_URL': 'https://ci.example',
+ 'TEAMCITY_CURRENT_TOKEN': 'test-current-token',
'GITHUB_TOKEN': 'test-github-token', 'FIXTURE_ROOT': str(self.root),
+ 'GITHUB_EVENT_PATH': str(self.root / 'event.json')}
+
+ def save(self, value):
+ (self.root / 'state.json').write_text(json.dumps(value))
+
+ def state(self):
+ return json.loads((self.root / 'state.json').read_text())
+
+ def run_shell(self, command, mock=True, check=True, env=None):
+ prefix = 'source "$1"\n'
+ if mock:
+ prefix += '''teamcity_request(){ python3 "$FIXTURE_ROOT/mock.py"
"$@"; }
+teamcity_current_pr(){ [[ ${PR_FRESH:-true} == true ]]; }
+curl(){ python3 "$FIXTURE_ROOT/status_mock.py" "$@"; }
+'''
+ result = subprocess.run(['bash', '-c', prefix + command, 'test',
str(HELPER)],
+ env={**self.env, **(env or {})}, text=True,
capture_output=True)
+ if check:
+ self.assertEqual(result.returncode, 0, result.stdout +
result.stderr)
+ return result
+
+ def trigger(self, lane='cloudut', count=1, **kwargs):
+ return self.run_shell('trigger_or_skip_build true 123 master ' + HEAD
+ ' ' + lane + ' ' + str(count), **kwargs)
+
+ def skipped_contexts(self):
+ p = self.root / 'statuses.json'
+ return [row['context'] for row in json.loads(p.read_text())] if
p.exists() else []
+
+ def test_unrelated_files_skip_without_submitting(self):
+ self.run_shell('trigger_or_skip_build false 123 master ' + HEAD + '
feut 1')
+ self.assertEqual(self.skipped_contexts(), ['TeamCity/feut'])
+ self.assertFalse(self.state().get('posts'))
+
+ def test_compile_skip_covers_downstream_checks(self):
+ self.run_shell('trigger_or_skip_build false 123 master ' + HEAD + '
compile 1')
+ self.assertEqual(self.skipped_contexts(), ['TeamCity/compile',
'TeamCity/p0', 'TeamCity/cloud_p0',
+ 'TeamCity/vault_p0',
'TeamCity/external', 'TeamCity/nonconcurrent'])
+ self.assertFalse(self.state().get('posts'))
+
+ def test_failed_skip_and_stale_head_do_not_succeed(self):
+ for env in [{'STATUS_FAIL': 'true'}, {'PR_FRESH': 'false'}]:
+ result = self.run_shell('trigger_or_skip_build false 123 master '
+ HEAD + ' feut 1', check=False, env=env)
+ self.assertNotEqual(result.returncode, 0)
+ self.assertFalse(self.skipped_contexts())
+
+ def file_filter_outputs(self, files, failed=False):
+ workflow = (HELPER.parents[3] /
'.github/workflows/comment-to-trigger-teamcity.yml').read_text()
+ block = workflow.split(' - name: "Check if pr need run build"\n',
1)[1].split(' # - name:', 1)[0]
+ script = textwrap.dedent(block.split(' run: |\n', 1)[1])
+ github_helper = HELPER.with_name('github-utils.sh')
+ override = 'source "' + str(github_helper) + '"\n'
+ override += '_get_pr_changed_files(){ return 1; }' if failed else
'_get_pr_changed_files(){ return 0; }'
+ script = script.replace('source
regression-test/pipeline/common/github-utils.sh', override)
+ script = script.replace('${{ steps.parse.outputs.PULL_REQUEST_NUM }}',
'123')
+ for name in ['all_files', 'modified_files']:
+ (self.root / name).write_text('\n'.join(files) + '\n')
+ for name in ['added_files', 'removed_files']:
+ (self.root / name).write_text('')
+ output = self.root / 'filter-output'; output.write_text('')
+ result = subprocess.run(['bash', '-eo', 'pipefail', '-c', script],
cwd=self.root,
+ env={**self.env, 'GITHUB_OUTPUT':
str(output)}, text=True, capture_output=True)
+ self.assertEqual(result.returncode, 0, result.stdout + result.stderr)
+ return dict(line.split('=', 1) for line in
output.read_text().splitlines())
+
+ def test_original_file_filters_and_workflow_wiring(self):
+ for files, expected in [(['fe/fe-core/src/main/java/Test.java'],
{'feut', 'p0', 'cloud_p0', 'vault_p0', 'external', 'nonconcurrent',
'performance', 'compile'}),
+ (['cloud/src/meta-service/meta_service.cpp'],
{'cloudut', 'cloud_p0', 'vault_p0', 'compile'}),
+ (['README.md'], set()),
+
(['regression-test/pipeline/p0/conf/regression-conf.groovy'], set())]:
+ flags = self.file_filter_outputs(files)
+ workflow = (HELPER.parents[3] /
'.github/workflows/comment-to-trigger-teamcity.yml').read_text()
+ selected = set()
+ for lane in ['compile', 'feut', 'beut', 'cloudut', 'p0',
'cloud_p0', 'vault_p0', 'external', 'nonconcurrent', 'performance']:
+ block = workflow.split(' - name: "Trigger ' + lane + '"\n',
1)[1].split(' - name:', 1)[0]
+ script = textwrap.dedent(block.split(' run: |\n', 1)[1])
+ script = script.replace('source
./regression-test/pipeline/common/teamcity-utils.sh', 'trigger_or_skip_build(){
echo "SELECTED=$1"; }')
+ values = {'COMMENT_TRIGGER_TYPE': lane, 'PULL_REQUEST_NUM':
'123', 'TARGET_BRANCH': 'master',
+ 'COMMIT_ID_FROM_TRIGGER': HEAD,
'COMMENT_REPEAT_TIMES': '1'}
+ for key, value in flags.items(): script = script.replace('${{
steps.changes.outputs.' + key + ' }}', value)
+ for key, value in values.items(): script = script.replace('${{
steps.parse.outputs.' + key + ' }}', value)
+ result = subprocess.run(['bash', '-eo', 'pipefail', '-c',
script], text=True, capture_output=True)
+ self.assertEqual(result.returncode, 0, result.stdout +
result.stderr)
+ if 'SELECTED=true' in result.stdout: selected.add(lane)
+ self.assertEqual(selected, expected)
+
+ def test_file_lookup_does_not_trace_github_token(self):
+ helper = HELPER.with_name('github-utils.sh')
+ script = 'source "$1"\n' + '''curl(){
+ if [[ "$*" == *--version* ]]; then return 0;
+ elif [[ "$*" == *'/files?'* ]]; then echo
'[{"filename":"README.md","status":"modified"}]';
+ else echo '{"changed_files":1}'; fi
+ }
+ _get_pr_changed_files 123
+'''
+ result = subprocess.run(['bash', '-c', script, 'test', str(helper)],
cwd=self.root,
+ env=self.env, text=True, capture_output=True)
+ self.assertEqual(result.returncode, 0, result.stdout + result.stderr)
+ self.assertNotIn('test-github-token', result.stdout + result.stderr)
+
+ def test_failed_file_lookup_does_not_select_skip(self):
+ flags = self.file_filter_outputs([], failed=True)
+ self.assertTrue(flags and all(value == 'true' for value in
flags.values()))
+
+ def test_keywords_match_required_checks(self):
+ repo = HELPER.parents[3]
+ required = set(re.findall(r'- (TeamCity/\S+)', (repo /
'.asf.yaml').read_text()))
+ result = self.run_shell('for lane in "${!comment_to_pipeline[@]}"; do
echo "$lane ${conment_to_context[$lane]}"; done')
+ mappings = dict(line.split() for line in result.stdout.splitlines())
+ self.assertEqual(set(mappings.values()), required |
{'TeamCity/external', 'TeamCity/performance'})
+ self.assertTrue(all(context == 'TeamCity/' + lane for lane, context in
mappings.items()))
+ workflow = (repo /
'.github/workflows/comment-to-trigger-teamcity.yml').read_text()
+ commands = re.search(r'reg="\^run \(([^)]+)\)',
workflow).group(1).split('|')
+ self.assertEqual(set(commands), set(mappings) | {'buildall'})
+
+ def parse_comment(self, command, branch='master', permission='write',
check=True, user_id='27881198'):
+ workflow = (HELPER.parents[3] /
'.github/workflows/comment-to-trigger-teamcity.yml').read_text()
+ block = workflow.split(' - name: "Parse PR comment"\n',
1)[1].split(' - name:', 1)[0]
+ script = textwrap.dedent(block.split(' run: |\n', 1)[1])
+ script = script.replace('${{ github.event.issue.pull_request.url }}',
'https://api.github.com/repos/apache/doris/pulls/123')
+ script = script.replace('${{ github.repository }}',
'apache/doris').replace('${{ github.event.comment.user.login }}', 'tester')
+ output = self.root / 'workflow-output'; output.write_text('')
+ prefix = 'curl(){ printf "%s\\n" "$MOCK_GITHUB_RESPONSE"; };\n'
+ result = subprocess.run(['bash', '-eo', 'pipefail', '-c', prefix +
script], text=True, capture_output=True,
+ env={**self.env, 'COMMENT_BODY': command,
'COMMENT_USER_ID': user_id,
+ 'GITHUB_TOKEN': 'test-github-token',
'GITHUB_OUTPUT': str(output),
+ 'GITHUB_ENV': str(self.root /
'workflow-env'),
+ 'MOCK_GITHUB_RESPONSE':
json.dumps({'head': {'sha': HEAD}, 'base': {'ref': branch}, 'permission':
permission})})
+ if check:
+ self.assertEqual(result.returncode, 0, result.stdout +
result.stderr)
+ return result, output.read_text()
+
+ def test_workflow_parses_exact_check_keywords(self):
+ for lane in ['compile', 'feut', 'beut', 'cloudut', 'p0', 'cloud_p0',
'vault_p0',
+ 'external', 'nonconcurrent', 'performance', 'buildall']:
+ result, output = self.parse_comment('run ' + lane + ' 2')
+ self.assertIn('COMMENT_TRIGGER_TYPE=' + lane + '\n', output)
+ self.assertIn('COMMENT_REPEAT_TIMES=2\n', output)
+ for command in ['run feut-extra', 'please run feut', 'run
nonConcurrent', 'run p1', 'run cloud_p1', 'run check_coverage']:
+ result, output = self.parse_comment(command, check=False)
+ self.assertNotIn('COMMENT_TRIGGER_TYPE=', output)
+
+ def test_skip_buildall_authorization_is_preserved(self):
+ _, output = self.parse_comment('skip buildall')
+ self.assertIn('comment_skip=true', output)
+ _, output = self.parse_comment('skip buildall', user_id='123456',
check=False)
+ self.assertNotIn('comment_skip=true', output)
+ _, output = self.parse_comment('skip buildall', branch='branch-4.2',
user_id='9208457')
+ self.assertIn('comment_skip=true', output)
+
+ def test_skip_buildall_statuses(self):
+ workflow = (HELPER.parents[3] /
'.github/workflows/comment-to-trigger-teamcity.yml').read_text()
+ block = workflow.split(' - name: "Skip buildall"\n', 1)[1]
+ script = textwrap.dedent(block.split(' run: |\n', 1)[1])
+ script = script.replace('source
./regression-test/pipeline/common/teamcity-utils.sh', ':')
+ script = script.replace('${{
steps.parse.outputs.COMMIT_ID_FROM_TRIGGER }}', HEAD)
+ for branch in ['master', 'branch-4.2', 'branch-4.1']:
+ (self.root / 'statuses.json').write_text('[]')
+ self.run_shell(script, env={'TARGET_BRANCH': branch})
+ expected = {'TeamCity/' + lane for lane in ['compile', 'feut',
'beut', 'cloudut', 'p0', 'cloud_p0', 'vault_p0', 'external', 'nonconcurrent']}
+ if branch == 'master': expected.add('TeamCity/performance')
+ self.assertEqual(set(self.skipped_contexts()), expected)
+ self.assertFalse(self.state().get('posts'))
+
+ def test_workflow_branch_and_permission_checks(self):
+ for branch in ['branch-4.2', 'branch-4.1']:
+ result, _ = self.parse_comment('run performance', branch=branch,
check=False)
+ self.assertNotEqual(result.returncode, 0)
+ self.parse_comment('run buildall', branch=branch)
+ self.parse_comment('run feut', branch=branch)
+ for branch, permission in [('branch-4.0', 'write'), ('master',
'read')]:
+ result, _ = self.parse_comment('run feut', branch=branch,
permission=permission, check=False)
+ self.assertNotEqual(result.returncode, 0)
+
+ def test_credentials_are_not_traced(self):
+ result = self.run_shell('''curl(){ printf '%s\\n' "$@" >
"$FIXTURE_ROOT/curl-args"; }
+set -x
+teamcity_request --fail https://ci.example
+''', mock=False)
+ self.assertNotIn('test-current-token', result.stdout + result.stderr)
+ args = (self.root / 'curl-args').read_text()
+ self.assertIn('Authorization: Bearer test-current-token', args)
+ self.assertNotIn('--user', args)
+
+ def test_retired_branch_has_no_submission(self):
+ result = self.run_shell('trigger_or_skip_build true 123 branch-4.0 ' +
HEAD + ' feut 1', check=False)
+ self.assertNotEqual(result.returncode, 0)
+ self.assertFalse(self.state().get('posts'))
+
+ def test_release_branches_reject_performance(self):
+ for branch in ['branch-4.2', 'branch-4.1']:
+ result = self.run_shell('trigger_or_skip_build true 123 ' + branch
+ ' ' + HEAD + ' performance 1', check=False)
+ self.assertNotEqual(result.returncode, 0)
+ self.assertFalse(self.state().get('posts'))
+
+ def test_status_token_is_not_traced(self):
+ result = self.run_shell('curl(){ return 0; }; set -x; skip_build
"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa" feut',
+ mock=False, env={'GITHUB_TOKEN':
'test-github-token'})
+ self.assertNotIn('test-github-token', result.stdout + result.stderr)
+
+ def test_missing_private_token_has_no_public_fallback(self):
+ result = self.run_shell('curl(){ exit 99; }; teamcity_request
https://ci.example', mock=False,
+ check=False, env={'TEAMCITY_CURRENT_TOKEN':
''})
+ self.assertEqual(result.returncode, 1)
+
+ def test_regressions_submit_directly_with_only_pr_parameters(self):
+ for lane, pipeline in [('p0', 'Doris_P0'), ('cloud_p0',
'Doris_CloudP0'),
+ ('vault_p0', 'Doris_VaultP0'), ('external',
'Doris_External'),
+ ('nonconcurrent', 'Doris_NonConcurrent')]:
+ self.save({})
+ self.trigger(lane)
+ body = self.state()['posts'][0]
+ self.assertEqual(body['buildType']['id'], pipeline)
+ self.assertEqual({p['name']: p['value'] for p in
body['properties']['property']},
+ {'env.PR_NUMBER': '123', 'env.CI_BRANCH':
'master'})
+ self.assertNotIn('custom-artifact-dependencies', body)
+ self.assertEqual(len(self.state()['posts']), 1)
+
+ def test_exact_revision_submission(self):
+ self.trigger()
+ body = self.state()['posts'][0]
+ self.assertEqual(body['buildType']['id'], 'Doris_CloudUT')
+ self.assertEqual(body['branchName'], 'pull/123')
+ self.assertEqual(body['properties']['property'][0]['name'],
'env.PR_NUMBER')
+
+ def test_active_build_does_not_prevent_submission(self):
+ self.save({'builds': [build()]})
+ self.trigger()
+ self.assertEqual(len(self.state()['posts']), 1)
+ self.assertEqual(self.state()['builds'][0]['state'], 'running')
+
+ def test_other_revision_is_preserved_and_not_reused(self):
+ self.save({'builds': [build(head='f' * 40)]})
+ self.trigger()
+ self.assertEqual(len(self.state()['posts']), 1)
+ self.assertEqual(self.state()['builds'][0]['state'], 'running')
+
+ def test_each_request_is_submitted_to_teamcity(self):
+ self.trigger(count=2)
+ self.trigger(count=2)
+ self.assertEqual(len(self.state()['posts']), 4)
+ for body in self.state()['posts']:
+ self.assertNotIn('env.CI_COMMENT_REQUEST_KEY', {p['name'] for p in
body['properties']['property']})
+
+ def test_uncertain_post_does_not_automatically_retry(self):
+ self.save({'lose_response': True})
+ self.assertNotEqual(self.trigger(check=False).returncode, 0)
+ self.assertEqual(len(self.state()['posts']), 1)
+
+ def test_changed_pr_does_not_submit(self):
+ self.assertNotEqual(self.trigger(check=False, env={'PR_FRESH':
'false'}).returncode, 0)
+ self.assertFalse(self.state().get('posts'))
+
+ def test_temporary_configuration_rejected(self):
+ self.save({'root': 'Doris_Performance_Schema70411_Debug'})
+ self.assertNotEqual(self.trigger('performance',
check=False).returncode, 0)
+ self.assertFalse(self.state().get('posts'))
+
+ def test_invalid_count_rejected(self):
+ for count in [0, 21, 'abc']:
+ self.assertNotEqual(self.trigger(count=count,
check=False).returncode, 0)
+ self.assertFalse(self.state().get('posts'))
+
+ def test_removed_lane_has_no_legacy_fallback(self):
+ self.assertNotEqual(self.trigger('p1', check=False).returncode, 0)
+ self.assertFalse(self.state().get('posts'))
+
+
+if __name__ == '__main__':
+ unittest.main()
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
