github-advanced-security[bot] commented on code in PR #5841: URL: https://github.com/apache/datafusion-comet/pull/5841#discussion_r3982217446
########## .github/workflows/pr_build_linux_checks.yml: ########## @@ -0,0 +1,262 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +name: PR Checks (Linux) + +# Independent lint, compilation, compatibility, and Rust debug checks. +# Run alongside the shared native producer; these jobs do not consume its artifact. +# Triggering and path filters live in the umbrella workflow. +on: + workflow_call: + +env: + RUST_VERSION: stable + RUST_BACKTRACE: 1 + # Force GNU ld on Linux: recent Rust stable defaults to rust-lld on + # x86_64-unknown-linux-gnu, and rust-lld cannot resolve -ljvm against the + # Zulu JDK layout installed by setup-java. Keep bfd for all cargo invocations. + RUSTFLAGS: "-Clink-arg=-fuse-ld=bfd" + +jobs: + + # Rust formatting gates the build and test jobs below + lint: + name: Lint + runs-on: ubuntu-24.04 + container: + image: amd64/rust + steps: + - uses: actions/checkout@v7 + + - name: Check Rust formatting + run: | + rustup component add rustfmt + cd native && cargo fmt --all -- --check + + # Fast syntactic-only scalafix check. Parses sources without compiling, so it + # surfaces version-independent style issues (e.g. RedundantSyntax) in seconds, + # long before the lint-java matrix finishes its ~3.5 min build. It also scans + # the spark-4.1 / spark-4.2 sources that lint-java skips (those profiles can't + # run -Psemanticdb yet), so it is the only gate covering them. The full rule + # set, including the semantic rules, still runs in lint-java. + scalafix-syntactic: + name: Lint Scala (syntactic) + runs-on: ubuntu-24.04 + steps: + - uses: actions/checkout@v7 + + - name: Setup coursier + uses: coursier/setup-action@v3 + with: + jvm: temurin:21 + + - name: Run syntactic scalafix check (no compile) + run: | + cs launch scalafix:0.14.6 -- \ + --check \ + --syntactic \ + --config .scalafix-syntactic.conf \ + --exclude '**/target/**' \ + spark + + lint-java: Review Comment: ## CodeQL / Workflow does not contain permissions Actions job or workflow does not limit the permissions of the GITHUB_TOKEN. Consider setting an explicit permissions block, using the following as a minimal starting point: {{contents: read}} [Show more details](https://github.com/apache/datafusion-comet/security/code-scanning/150) ########## .github/workflows/ci.yml: ########## @@ -179,16 +182,112 @@ # each time somebody applies a label. # --------------------------------------------------------------------------- + # Independent checks start immediately after change selection. + pr_build_linux_checks: + name: PR Checks (Linux) + needs: changes + if: | + needs.changes.outputs.build_linux == 'true' && + (github.event_name == 'push' || + github.event_name == 'workflow_dispatch' || + (github.event_name == 'pull_request' && + github.event.action != 'labeled')) + uses: ./.github/workflows/pr_build_linux_checks.yml + + # Build once for the union of the native-library consumers selected below. + # Preserve their event/label gates: a legacy-version diff without its opt-in + # label, or an unrelated label event, must not start an unused native build. + build_linux_native: Review Comment: ## CodeQL / Workflow does not contain permissions Actions job or workflow does not limit the permissions of the GITHUB_TOKEN. Consider setting an explicit permissions block, using the following as a minimal starting point: {{}} [Show more details](https://github.com/apache/datafusion-comet/security/code-scanning/148) ########## .github/workflows/pr_build_linux_checks.yml: ########## @@ -0,0 +1,262 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +name: PR Checks (Linux) + +# Independent lint, compilation, compatibility, and Rust debug checks. +# Run alongside the shared native producer; these jobs do not consume its artifact. +# Triggering and path filters live in the umbrella workflow. +on: + workflow_call: + +env: + RUST_VERSION: stable + RUST_BACKTRACE: 1 + # Force GNU ld on Linux: recent Rust stable defaults to rust-lld on + # x86_64-unknown-linux-gnu, and rust-lld cannot resolve -ljvm against the + # Zulu JDK layout installed by setup-java. Keep bfd for all cargo invocations. + RUSTFLAGS: "-Clink-arg=-fuse-ld=bfd" + +jobs: + + # Rust formatting gates the build and test jobs below + lint: + name: Lint + runs-on: ubuntu-24.04 + container: + image: amd64/rust + steps: + - uses: actions/checkout@v7 + + - name: Check Rust formatting + run: | + rustup component add rustfmt + cd native && cargo fmt --all -- --check + + # Fast syntactic-only scalafix check. Parses sources without compiling, so it + # surfaces version-independent style issues (e.g. RedundantSyntax) in seconds, + # long before the lint-java matrix finishes its ~3.5 min build. It also scans + # the spark-4.1 / spark-4.2 sources that lint-java skips (those profiles can't + # run -Psemanticdb yet), so it is the only gate covering them. The full rule + # set, including the semantic rules, still runs in lint-java. + scalafix-syntactic: + name: Lint Scala (syntactic) + runs-on: ubuntu-24.04 + steps: + - uses: actions/checkout@v7 + + - name: Setup coursier + uses: coursier/setup-action@v3 + with: + jvm: temurin:21 + + - name: Run syntactic scalafix check (no compile) + run: | + cs launch scalafix:0.14.6 -- \ + --check \ + --syntactic \ + --config .scalafix-syntactic.conf \ + --exclude '**/target/**' \ + spark + + lint-java: + needs: lint + name: Lint Java (${{ matrix.profile.name }}) + runs-on: ubuntu-24.04 + container: + image: amd64/rust + env: + JAVA_TOOL_OPTIONS: ${{ (matrix.profile.java_version == '17' || matrix.profile.java_version == '21') && '--add-exports=java.base/sun.nio.ch=ALL-UNNAMED --add-exports=java.base/sun.util.calendar=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/java.lang=ALL-UNNAMED' || '' }} + strategy: + matrix: + profile: + - name: "Spark 3.4, JDK 11, Scala 2.12" + java_version: "11" + maven_opts: "-Pspark-3.4 -Pscala-2.12" + - name: "Spark 3.5, JDK 17, Scala 2.12" + java_version: "17" + maven_opts: "-Pspark-3.5 -Pscala-2.12" + - name: "Spark 4.0, JDK 17" + java_version: "17" + maven_opts: "-Pspark-4.0" + - name: "Spark 4.0, JDK 21" + java_version: "21" + maven_opts: "-Pspark-4.0" + # Spark 4.1 and 4.2 are intentionally absent: the lint job invokes -Psemanticdb, + # but semanticdb-scalac for those Scala patch versions (2.13.17 / 2.13.18) is not + # yet published, so we cannot currently run scalafix against the spark-4.1 or + # spark-4.2 profiles. + fail-fast: false + steps: + - uses: actions/checkout@v7 + + - name: Setup Rust & Java toolchain + uses: ./.github/actions/setup-builder + with: + rust-version: ${{ env.RUST_VERSION }} + jdk-version: ${{ matrix.profile.java_version }} + + - name: Cache Maven dependencies + uses: actions/cache@v6 + with: + path: | + ~/.m2/repository + /root/.m2/repository + key: ${{ runner.os }}-java-maven-${{ hashFiles('**/pom.xml') }}-lint + restore-keys: | + ${{ runner.os }}-java-maven- + + - name: Run scalafix check + run: | + ./mvnw -B package -DskipTests scalafix:scalafix -Dscalafix.mode=CHECK -Psemanticdb ${{ matrix.profile.maven_opts }} + + - name: Setup Node.js + uses: actions/setup-node@v7 + with: + node-version: '24' + + - name: Install prettier + run: | + npm install -g prettier + + - name: Run prettier + run: | + npx prettier "**/*.md" --write + + - name: Mark workspace as safe for git + run: | + git config --global --add safe.directory "$GITHUB_WORKSPACE" + + - name: Check for any local git changes (such as generated docs) + run: | + ./dev/ci/check-working-tree-clean.sh + + # Compile-only verification for Spark 4.1. Tests are intentionally skipped: the spark-4.1 + # profile is currently a build target only, and several runtime/test failures are tracked + # in follow-up PRs. Excluded from lint-java because semanticdb-scalac_2.13.17 is not yet + # published and the lint job activates -Psemanticdb. + build-spark-4-1: + needs: lint + name: Build Spark 4.1, JDK 17 + runs-on: ubuntu-24.04 + container: + image: amd64/rust + steps: + - uses: actions/checkout@v7 + + - name: Setup Rust & Java toolchain + uses: ./.github/actions/setup-builder + with: + rust-version: ${{ env.RUST_VERSION }} + jdk-version: 17 + + - name: Cache Maven dependencies + uses: actions/cache@v6 + with: + path: | + ~/.m2/repository + /root/.m2/repository + key: ${{ runner.os }}-java-maven-${{ hashFiles('**/pom.xml') }}-spark-4.1-build + restore-keys: | + ${{ runner.os }}-java-maven- + + - name: Compile (skip tests) + run: ./mvnw -B install -DskipTests -Dmaven.test.skip=true -Pspark-4.1 + + celeborn-reflection-compatibility: + needs: lint + name: Celeborn ${{ matrix.celeborn_version }} reflection compatibility + runs-on: ubuntu-24.04 + container: + image: amd64/rust + env: + JAVA_TOOL_OPTIONS: --add-exports=java.base/sun.nio.ch=ALL-UNNAMED --add-exports=java.base/sun.util.calendar=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/java.lang=ALL-UNNAMED + strategy: + fail-fast: false + matrix: + celeborn_version: ["0.6.0", "0.7.0"] + steps: + - uses: actions/checkout@v7 + + - name: Setup Rust & Java toolchain + uses: ./.github/actions/setup-builder + with: + rust-version: ${{ env.RUST_VERSION }} + jdk-version: 17 + + - name: Cache Maven dependencies + uses: actions/cache@v6 + with: + path: | + ~/.m2/repository + /root/.m2/repository + key: ${{ runner.os }}-java-maven-${{ hashFiles('**/pom.xml') }}-celeborn-${{ matrix.celeborn_version }} + restore-keys: | + ${{ runner.os }}-java-maven- + + - name: Verify reflected Celeborn internals + env: + SPARK_LOCAL_HOSTNAME: localhost + SPARK_LOCAL_IP: 127.0.0.1 + run: | + SPARK_HOME="$GITHUB_WORKSPACE" ./mvnw -B clean test \ + -Pspark-3.5,scala-2.12,celeborn-reflection-compatibility \ + -Dceleborn.version="${{ matrix.celeborn_version }}" \ + -Dtest=none \ + -Dsuites=org.apache.comet.shuffle.CelebornReflectionCompatibilitySuite \ + -DfailIfNoTests=false + + # Rust tests use a separate debug build, not the shared CI library. + linux-test-rust: Review Comment: ## CodeQL / Workflow does not contain permissions Actions job or workflow does not limit the permissions of the GITHUB_TOKEN. Consider setting an explicit permissions block, using the following as a minimal starting point: {{contents: read}} [Show more details](https://github.com/apache/datafusion-comet/security/code-scanning/153) ########## .github/workflows/pr_build_linux_checks.yml: ########## @@ -0,0 +1,262 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +name: PR Checks (Linux) + +# Independent lint, compilation, compatibility, and Rust debug checks. +# Run alongside the shared native producer; these jobs do not consume its artifact. +# Triggering and path filters live in the umbrella workflow. +on: + workflow_call: + +env: + RUST_VERSION: stable + RUST_BACKTRACE: 1 + # Force GNU ld on Linux: recent Rust stable defaults to rust-lld on + # x86_64-unknown-linux-gnu, and rust-lld cannot resolve -ljvm against the + # Zulu JDK layout installed by setup-java. Keep bfd for all cargo invocations. + RUSTFLAGS: "-Clink-arg=-fuse-ld=bfd" + +jobs: + + # Rust formatting gates the build and test jobs below + lint: + name: Lint + runs-on: ubuntu-24.04 + container: + image: amd64/rust + steps: + - uses: actions/checkout@v7 + + - name: Check Rust formatting + run: | + rustup component add rustfmt + cd native && cargo fmt --all -- --check + + # Fast syntactic-only scalafix check. Parses sources without compiling, so it + # surfaces version-independent style issues (e.g. RedundantSyntax) in seconds, + # long before the lint-java matrix finishes its ~3.5 min build. It also scans + # the spark-4.1 / spark-4.2 sources that lint-java skips (those profiles can't + # run -Psemanticdb yet), so it is the only gate covering them. The full rule + # set, including the semantic rules, still runs in lint-java. + scalafix-syntactic: + name: Lint Scala (syntactic) + runs-on: ubuntu-24.04 + steps: + - uses: actions/checkout@v7 + + - name: Setup coursier + uses: coursier/setup-action@v3 + with: + jvm: temurin:21 + + - name: Run syntactic scalafix check (no compile) + run: | + cs launch scalafix:0.14.6 -- \ + --check \ + --syntactic \ + --config .scalafix-syntactic.conf \ + --exclude '**/target/**' \ + spark + + lint-java: + needs: lint + name: Lint Java (${{ matrix.profile.name }}) + runs-on: ubuntu-24.04 + container: + image: amd64/rust + env: + JAVA_TOOL_OPTIONS: ${{ (matrix.profile.java_version == '17' || matrix.profile.java_version == '21') && '--add-exports=java.base/sun.nio.ch=ALL-UNNAMED --add-exports=java.base/sun.util.calendar=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/java.lang=ALL-UNNAMED' || '' }} + strategy: + matrix: + profile: + - name: "Spark 3.4, JDK 11, Scala 2.12" + java_version: "11" + maven_opts: "-Pspark-3.4 -Pscala-2.12" + - name: "Spark 3.5, JDK 17, Scala 2.12" + java_version: "17" + maven_opts: "-Pspark-3.5 -Pscala-2.12" + - name: "Spark 4.0, JDK 17" + java_version: "17" + maven_opts: "-Pspark-4.0" + - name: "Spark 4.0, JDK 21" + java_version: "21" + maven_opts: "-Pspark-4.0" + # Spark 4.1 and 4.2 are intentionally absent: the lint job invokes -Psemanticdb, + # but semanticdb-scalac for those Scala patch versions (2.13.17 / 2.13.18) is not + # yet published, so we cannot currently run scalafix against the spark-4.1 or + # spark-4.2 profiles. + fail-fast: false + steps: + - uses: actions/checkout@v7 + + - name: Setup Rust & Java toolchain + uses: ./.github/actions/setup-builder + with: + rust-version: ${{ env.RUST_VERSION }} + jdk-version: ${{ matrix.profile.java_version }} + + - name: Cache Maven dependencies + uses: actions/cache@v6 + with: + path: | + ~/.m2/repository + /root/.m2/repository + key: ${{ runner.os }}-java-maven-${{ hashFiles('**/pom.xml') }}-lint + restore-keys: | + ${{ runner.os }}-java-maven- + + - name: Run scalafix check + run: | + ./mvnw -B package -DskipTests scalafix:scalafix -Dscalafix.mode=CHECK -Psemanticdb ${{ matrix.profile.maven_opts }} + + - name: Setup Node.js + uses: actions/setup-node@v7 + with: + node-version: '24' + + - name: Install prettier + run: | + npm install -g prettier + + - name: Run prettier + run: | + npx prettier "**/*.md" --write + + - name: Mark workspace as safe for git + run: | + git config --global --add safe.directory "$GITHUB_WORKSPACE" + + - name: Check for any local git changes (such as generated docs) + run: | + ./dev/ci/check-working-tree-clean.sh + + # Compile-only verification for Spark 4.1. Tests are intentionally skipped: the spark-4.1 + # profile is currently a build target only, and several runtime/test failures are tracked + # in follow-up PRs. Excluded from lint-java because semanticdb-scalac_2.13.17 is not yet + # published and the lint job activates -Psemanticdb. + build-spark-4-1: + needs: lint + name: Build Spark 4.1, JDK 17 + runs-on: ubuntu-24.04 + container: + image: amd64/rust + steps: + - uses: actions/checkout@v7 + + - name: Setup Rust & Java toolchain + uses: ./.github/actions/setup-builder + with: + rust-version: ${{ env.RUST_VERSION }} + jdk-version: 17 + + - name: Cache Maven dependencies + uses: actions/cache@v6 + with: + path: | + ~/.m2/repository + /root/.m2/repository + key: ${{ runner.os }}-java-maven-${{ hashFiles('**/pom.xml') }}-spark-4.1-build + restore-keys: | + ${{ runner.os }}-java-maven- + + - name: Compile (skip tests) + run: ./mvnw -B install -DskipTests -Dmaven.test.skip=true -Pspark-4.1 + + celeborn-reflection-compatibility: + needs: lint + name: Celeborn ${{ matrix.celeborn_version }} reflection compatibility + runs-on: ubuntu-24.04 + container: + image: amd64/rust + env: + JAVA_TOOL_OPTIONS: --add-exports=java.base/sun.nio.ch=ALL-UNNAMED --add-exports=java.base/sun.util.calendar=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/java.lang=ALL-UNNAMED + strategy: + fail-fast: false + matrix: + celeborn_version: ["0.6.0", "0.7.0"] + steps: + - uses: actions/checkout@v7 + + - name: Setup Rust & Java toolchain + uses: ./.github/actions/setup-builder + with: + rust-version: ${{ env.RUST_VERSION }} + jdk-version: 17 + + - name: Cache Maven dependencies + uses: actions/cache@v6 + with: + path: | + ~/.m2/repository + /root/.m2/repository + key: ${{ runner.os }}-java-maven-${{ hashFiles('**/pom.xml') }}-celeborn-${{ matrix.celeborn_version }} + restore-keys: | + ${{ runner.os }}-java-maven- + + - name: Verify reflected Celeborn internals + env: + SPARK_LOCAL_HOSTNAME: localhost + SPARK_LOCAL_IP: 127.0.0.1 + run: | + SPARK_HOME="$GITHUB_WORKSPACE" ./mvnw -B clean test \ + -Pspark-3.5,scala-2.12,celeborn-reflection-compatibility \ + -Dceleborn.version="${{ matrix.celeborn_version }}" \ + -Dtest=none \ + -Dsuites=org.apache.comet.shuffle.CelebornReflectionCompatibilitySuite \ + -DfailIfNoTests=false + + # Rust tests use a separate debug build, not the shared CI library. + linux-test-rust: + needs: lint + name: ubuntu-latest/rust-test + runs-on: ubuntu-24.04 + container: + image: amd64/rust + steps: + - uses: actions/checkout@v7 + + - name: Setup Rust & Java toolchain + uses: ./.github/actions/setup-builder + with: + rust-version: ${{ env.RUST_VERSION }} + jdk-version: 17 + + - name: Restore Cargo cache + uses: actions/cache/restore@v6 + with: + path: | + ~/.cargo/registry + ~/.cargo/git + native/target + # Note: Java version intentionally excluded - Rust target is JDK-independent + key: ${{ runner.os }}-cargo-debug-${{ hashFiles('native/**/Cargo.lock', 'native/**/Cargo.toml') }}-${{ hashFiles('native/**/*.rs') }} + restore-keys: | + ${{ runner.os }}-cargo-debug-${{ hashFiles('native/**/Cargo.lock', 'native/**/Cargo.toml') }}- + + - name: Rust test steps + uses: ./.github/actions/rust-test + + - name: Save Cargo cache + uses: actions/cache/save@v6 + if: github.ref == 'refs/heads/main' + with: + path: | + ~/.cargo/registry + ~/.cargo/git + native/target + key: ${{ runner.os }}-cargo-debug-${{ hashFiles('native/**/Cargo.lock', 'native/**/Cargo.toml') }}-${{ hashFiles('native/**/*.rs') }} Review Comment: ## CodeQL / Workflow does not contain permissions Actions job or workflow does not limit the permissions of the GITHUB_TOKEN. Consider setting an explicit permissions block, using the following as a minimal starting point: {{contents: read}} [Show more details](https://github.com/apache/datafusion-comet/security/code-scanning/154) ########## .github/workflows/pr_build_linux_checks.yml: ########## @@ -0,0 +1,262 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +name: PR Checks (Linux) + +# Independent lint, compilation, compatibility, and Rust debug checks. +# Run alongside the shared native producer; these jobs do not consume its artifact. +# Triggering and path filters live in the umbrella workflow. +on: + workflow_call: + +env: + RUST_VERSION: stable + RUST_BACKTRACE: 1 + # Force GNU ld on Linux: recent Rust stable defaults to rust-lld on + # x86_64-unknown-linux-gnu, and rust-lld cannot resolve -ljvm against the + # Zulu JDK layout installed by setup-java. Keep bfd for all cargo invocations. + RUSTFLAGS: "-Clink-arg=-fuse-ld=bfd" + +jobs: + + # Rust formatting gates the build and test jobs below + lint: + name: Lint + runs-on: ubuntu-24.04 + container: + image: amd64/rust + steps: + - uses: actions/checkout@v7 + + - name: Check Rust formatting + run: | + rustup component add rustfmt + cd native && cargo fmt --all -- --check + + # Fast syntactic-only scalafix check. Parses sources without compiling, so it + # surfaces version-independent style issues (e.g. RedundantSyntax) in seconds, + # long before the lint-java matrix finishes its ~3.5 min build. It also scans + # the spark-4.1 / spark-4.2 sources that lint-java skips (those profiles can't + # run -Psemanticdb yet), so it is the only gate covering them. The full rule + # set, including the semantic rules, still runs in lint-java. + scalafix-syntactic: + name: Lint Scala (syntactic) + runs-on: ubuntu-24.04 + steps: + - uses: actions/checkout@v7 + + - name: Setup coursier + uses: coursier/setup-action@v3 + with: + jvm: temurin:21 + + - name: Run syntactic scalafix check (no compile) + run: | + cs launch scalafix:0.14.6 -- \ + --check \ + --syntactic \ + --config .scalafix-syntactic.conf \ + --exclude '**/target/**' \ + spark + + lint-java: + needs: lint + name: Lint Java (${{ matrix.profile.name }}) + runs-on: ubuntu-24.04 + container: + image: amd64/rust + env: + JAVA_TOOL_OPTIONS: ${{ (matrix.profile.java_version == '17' || matrix.profile.java_version == '21') && '--add-exports=java.base/sun.nio.ch=ALL-UNNAMED --add-exports=java.base/sun.util.calendar=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/java.lang=ALL-UNNAMED' || '' }} + strategy: + matrix: + profile: + - name: "Spark 3.4, JDK 11, Scala 2.12" + java_version: "11" + maven_opts: "-Pspark-3.4 -Pscala-2.12" + - name: "Spark 3.5, JDK 17, Scala 2.12" + java_version: "17" + maven_opts: "-Pspark-3.5 -Pscala-2.12" + - name: "Spark 4.0, JDK 17" + java_version: "17" + maven_opts: "-Pspark-4.0" + - name: "Spark 4.0, JDK 21" + java_version: "21" + maven_opts: "-Pspark-4.0" + # Spark 4.1 and 4.2 are intentionally absent: the lint job invokes -Psemanticdb, + # but semanticdb-scalac for those Scala patch versions (2.13.17 / 2.13.18) is not + # yet published, so we cannot currently run scalafix against the spark-4.1 or + # spark-4.2 profiles. + fail-fast: false + steps: + - uses: actions/checkout@v7 + + - name: Setup Rust & Java toolchain + uses: ./.github/actions/setup-builder + with: + rust-version: ${{ env.RUST_VERSION }} + jdk-version: ${{ matrix.profile.java_version }} + + - name: Cache Maven dependencies + uses: actions/cache@v6 + with: + path: | + ~/.m2/repository + /root/.m2/repository + key: ${{ runner.os }}-java-maven-${{ hashFiles('**/pom.xml') }}-lint + restore-keys: | + ${{ runner.os }}-java-maven- + + - name: Run scalafix check + run: | + ./mvnw -B package -DskipTests scalafix:scalafix -Dscalafix.mode=CHECK -Psemanticdb ${{ matrix.profile.maven_opts }} + + - name: Setup Node.js + uses: actions/setup-node@v7 + with: + node-version: '24' + + - name: Install prettier + run: | + npm install -g prettier + + - name: Run prettier + run: | + npx prettier "**/*.md" --write + + - name: Mark workspace as safe for git + run: | + git config --global --add safe.directory "$GITHUB_WORKSPACE" + + - name: Check for any local git changes (such as generated docs) + run: | + ./dev/ci/check-working-tree-clean.sh + + # Compile-only verification for Spark 4.1. Tests are intentionally skipped: the spark-4.1 + # profile is currently a build target only, and several runtime/test failures are tracked + # in follow-up PRs. Excluded from lint-java because semanticdb-scalac_2.13.17 is not yet + # published and the lint job activates -Psemanticdb. + build-spark-4-1: Review Comment: ## CodeQL / Workflow does not contain permissions Actions job or workflow does not limit the permissions of the GITHUB_TOKEN. Consider setting an explicit permissions block, using the following as a minimal starting point: {{contents: read}} [Show more details](https://github.com/apache/datafusion-comet/security/code-scanning/151) ########## .github/workflows/pr_build_linux_checks.yml: ########## @@ -0,0 +1,262 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +name: PR Checks (Linux) + +# Independent lint, compilation, compatibility, and Rust debug checks. +# Run alongside the shared native producer; these jobs do not consume its artifact. +# Triggering and path filters live in the umbrella workflow. +on: + workflow_call: + +env: + RUST_VERSION: stable + RUST_BACKTRACE: 1 + # Force GNU ld on Linux: recent Rust stable defaults to rust-lld on + # x86_64-unknown-linux-gnu, and rust-lld cannot resolve -ljvm against the + # Zulu JDK layout installed by setup-java. Keep bfd for all cargo invocations. + RUSTFLAGS: "-Clink-arg=-fuse-ld=bfd" + +jobs: + + # Rust formatting gates the build and test jobs below + lint: + name: Lint + runs-on: ubuntu-24.04 + container: + image: amd64/rust + steps: + - uses: actions/checkout@v7 + + - name: Check Rust formatting + run: | + rustup component add rustfmt + cd native && cargo fmt --all -- --check + + # Fast syntactic-only scalafix check. Parses sources without compiling, so it + # surfaces version-independent style issues (e.g. RedundantSyntax) in seconds, + # long before the lint-java matrix finishes its ~3.5 min build. It also scans + # the spark-4.1 / spark-4.2 sources that lint-java skips (those profiles can't + # run -Psemanticdb yet), so it is the only gate covering them. The full rule + # set, including the semantic rules, still runs in lint-java. + scalafix-syntactic: + name: Lint Scala (syntactic) + runs-on: ubuntu-24.04 + steps: + - uses: actions/checkout@v7 + + - name: Setup coursier + uses: coursier/setup-action@v3 + with: + jvm: temurin:21 + + - name: Run syntactic scalafix check (no compile) + run: | + cs launch scalafix:0.14.6 -- \ + --check \ + --syntactic \ + --config .scalafix-syntactic.conf \ + --exclude '**/target/**' \ + spark + + lint-java: + needs: lint + name: Lint Java (${{ matrix.profile.name }}) + runs-on: ubuntu-24.04 + container: + image: amd64/rust + env: + JAVA_TOOL_OPTIONS: ${{ (matrix.profile.java_version == '17' || matrix.profile.java_version == '21') && '--add-exports=java.base/sun.nio.ch=ALL-UNNAMED --add-exports=java.base/sun.util.calendar=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/java.lang=ALL-UNNAMED' || '' }} + strategy: + matrix: + profile: + - name: "Spark 3.4, JDK 11, Scala 2.12" + java_version: "11" + maven_opts: "-Pspark-3.4 -Pscala-2.12" + - name: "Spark 3.5, JDK 17, Scala 2.12" + java_version: "17" + maven_opts: "-Pspark-3.5 -Pscala-2.12" + - name: "Spark 4.0, JDK 17" + java_version: "17" + maven_opts: "-Pspark-4.0" + - name: "Spark 4.0, JDK 21" + java_version: "21" + maven_opts: "-Pspark-4.0" + # Spark 4.1 and 4.2 are intentionally absent: the lint job invokes -Psemanticdb, + # but semanticdb-scalac for those Scala patch versions (2.13.17 / 2.13.18) is not + # yet published, so we cannot currently run scalafix against the spark-4.1 or + # spark-4.2 profiles. + fail-fast: false + steps: + - uses: actions/checkout@v7 + + - name: Setup Rust & Java toolchain + uses: ./.github/actions/setup-builder + with: + rust-version: ${{ env.RUST_VERSION }} + jdk-version: ${{ matrix.profile.java_version }} + + - name: Cache Maven dependencies + uses: actions/cache@v6 + with: + path: | + ~/.m2/repository + /root/.m2/repository + key: ${{ runner.os }}-java-maven-${{ hashFiles('**/pom.xml') }}-lint + restore-keys: | + ${{ runner.os }}-java-maven- + + - name: Run scalafix check + run: | + ./mvnw -B package -DskipTests scalafix:scalafix -Dscalafix.mode=CHECK -Psemanticdb ${{ matrix.profile.maven_opts }} + + - name: Setup Node.js + uses: actions/setup-node@v7 + with: + node-version: '24' + + - name: Install prettier + run: | + npm install -g prettier + + - name: Run prettier + run: | + npx prettier "**/*.md" --write + + - name: Mark workspace as safe for git + run: | + git config --global --add safe.directory "$GITHUB_WORKSPACE" + + - name: Check for any local git changes (such as generated docs) + run: | + ./dev/ci/check-working-tree-clean.sh + + # Compile-only verification for Spark 4.1. Tests are intentionally skipped: the spark-4.1 + # profile is currently a build target only, and several runtime/test failures are tracked + # in follow-up PRs. Excluded from lint-java because semanticdb-scalac_2.13.17 is not yet + # published and the lint job activates -Psemanticdb. + build-spark-4-1: + needs: lint + name: Build Spark 4.1, JDK 17 + runs-on: ubuntu-24.04 + container: + image: amd64/rust + steps: + - uses: actions/checkout@v7 + + - name: Setup Rust & Java toolchain + uses: ./.github/actions/setup-builder + with: + rust-version: ${{ env.RUST_VERSION }} + jdk-version: 17 + + - name: Cache Maven dependencies + uses: actions/cache@v6 + with: + path: | + ~/.m2/repository + /root/.m2/repository + key: ${{ runner.os }}-java-maven-${{ hashFiles('**/pom.xml') }}-spark-4.1-build + restore-keys: | + ${{ runner.os }}-java-maven- + + - name: Compile (skip tests) + run: ./mvnw -B install -DskipTests -Dmaven.test.skip=true -Pspark-4.1 + + celeborn-reflection-compatibility: Review Comment: ## CodeQL / Workflow does not contain permissions Actions job or workflow does not limit the permissions of the GITHUB_TOKEN. Consider setting an explicit permissions block, using the following as a minimal starting point: {{contents: read}} [Show more details](https://github.com/apache/datafusion-comet/security/code-scanning/152) ########## .github/workflows/pr_build_linux_checks.yml: ########## @@ -0,0 +1,262 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +name: PR Checks (Linux) + +# Independent lint, compilation, compatibility, and Rust debug checks. +# Run alongside the shared native producer; these jobs do not consume its artifact. +# Triggering and path filters live in the umbrella workflow. +on: + workflow_call: + +env: + RUST_VERSION: stable + RUST_BACKTRACE: 1 + # Force GNU ld on Linux: recent Rust stable defaults to rust-lld on + # x86_64-unknown-linux-gnu, and rust-lld cannot resolve -ljvm against the + # Zulu JDK layout installed by setup-java. Keep bfd for all cargo invocations. + RUSTFLAGS: "-Clink-arg=-fuse-ld=bfd" + +jobs: + + # Rust formatting gates the build and test jobs below + lint: + name: Lint + runs-on: ubuntu-24.04 + container: + image: amd64/rust + steps: + - uses: actions/checkout@v7 + + - name: Check Rust formatting + run: | + rustup component add rustfmt + cd native && cargo fmt --all -- --check + + # Fast syntactic-only scalafix check. Parses sources without compiling, so it + # surfaces version-independent style issues (e.g. RedundantSyntax) in seconds, + # long before the lint-java matrix finishes its ~3.5 min build. It also scans + # the spark-4.1 / spark-4.2 sources that lint-java skips (those profiles can't + # run -Psemanticdb yet), so it is the only gate covering them. The full rule + # set, including the semantic rules, still runs in lint-java. + scalafix-syntactic: Review Comment: ## CodeQL / Workflow does not contain permissions Actions job or workflow does not limit the permissions of the GITHUB_TOKEN. Consider setting an explicit permissions block, using the following as a minimal starting point: {{contents: read}} [Show more details](https://github.com/apache/datafusion-comet/security/code-scanning/149) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
