schenksj opened a new pull request, #4952: URL: https://github.com/apache/datafusion-comet/pull/4952
**Part 2 of the Delta Lake contrib PR breakup.** Part 1 (#4700, the core SPI for contrib leaf scans) is merged. This part establishes the `contrib-delta` **build gate** and the **inert wiring** that lets a gated build compile and link end to end — while the **default build stays byte-for-byte unchanged** (zero Delta surface). It ships **no real Delta read logic**: a Delta read that reaches native returns a clean "not implemented" error and falls back to vanilla Spark. The full sequence and dependency graph live in the tracking umbrella, #4366. The whole point of this part is that everything it adds is gated or inert, so it is safe to land on `main` well ahead of the read path, and reviewers can review the wire format, the build machinery, and the gate enforcement **once, in isolation**, before any Delta code shows up. ## Changes **Build gate** - **Maven** `contrib-delta` profile (`spark/pom.xml`) with a per-Spark `delta.version` (3.5 → 3.3.2, 4.0 → 4.0.0, 4.1 → 4.1.0) and an `add-source` of `contrib/delta/src`. Default `delta.version` floor in `pom.xml`. The default `spark.version` stays **4.1.2** — the delta-spark 4.1.1 pin is a separate, deferred decision (raised for later parts). - **Cargo** `contrib-delta` feature on core (optional path dep on `comet-contrib-delta`); `native/Cargo.toml` excludes `../contrib` from the workspace so non-Delta committers never build it. - **`dev/verify-contrib-delta-gate.sh`** proves the default cargo tree, Maven dependency set, compiled classes, and `libcomet` all carry **zero Delta surface**, and that the gated build pulls the right deps (delta-spark per Spark profile, `comet-contrib-delta` in the cargo tree). Wired into a minimal `delta_build_gate.yml` CI job. The full test-suite and regression workflows land in later parts. **Inert wiring** - **Proto:** `Delta*` messages and `delta_scan = 118` (117 is `BroadcastNestedLoopJoin`). One-time wire-format review here, early. - **Native dispatch:** an `OpStruct::DeltaScan` arm that returns a not-compiled-in error on default builds, and a feature-gated `delta_scan` shim that calls the contrib; exhaustive-match arms in `operator_registry` / `jni_api`; `convert_spark_types_to_arrow_schema` promoted to `pub(crate)`. - **Stub contrib crate** (`contrib/delta/native`): `plan_delta_scan` returns `DataFusionError::NotImplemented` — just enough to satisfy the core shim's contract so `--features contrib-delta` links. This makes the exact core↔contrib contract visible in a small PR. - **JVM bridge `DeltaIntegration`** (reflective; every lookup returns `None` until the contrib classes exist), the `CometExecRule` Delta-marker hook (the CDF hook is deferred to a later part), the `CometScanRule` Delta delegation + metadata-column reorder, and the leaf `DeltaConf`. ## What this part deliberately does NOT do yet - **No Rust read path** — `plan_delta_scan` is a stub. Log replay, predicate pushdown, deletion vectors, and the kernel read land in parts 3a/3b. - **No Scala claim/decline or execution** — `DeltaIntegration`'s lookups resolve to `None`, so the marker always falls back. The claim/decline layer and native exec land in parts 4a/4b. - **No CDF** — the `CometExecRule` CDF hook and `DeltaIntegration`'s CDF members are held back to part 5. ## Why it is safe on default builds Without `-Pcontrib-delta`: the cargo feature is off (no `comet-contrib-delta`, no `delta_kernel` in the tree), the Maven profile is inactive (no `io.delta:*`, no `contrib/delta` sources compiled), and the native `DeltaScan` arm is `#[cfg]`-stubbed to an error that is never reached because nothing emits the proto message. `DeltaIntegration` is the only always-present class and every one of its reflective lookups returns `None`. The gate script asserts all of this mechanically: default `libcomet` has **0 Delta symbols** and is the same size as before. The `CometScanRule` change reorders the metadata-column guard so V1 scans reach `transformV1Scan` (which delegates to any V1 contrib) *before* the generic metadata-column rejection — but for a non-contrib V1 scan the guard is re-applied inside `transformV1Scan`, so vanilla behavior is unchanged. ## Verification Run against this branch rebased on current `main`: - Default native build + gated native build (`--features contrib-delta`): green. - `cargo clippy` both feature states: clean. - `dev/verify-contrib-delta-gate.sh`: all checks pass (0 Delta symbols in the default dylib). - Gated JVM compile (spark-3.5 / Scala 2.13, `-Pcontrib-delta`) and default JVM compile (spark-3.4 / **Scala 2.12**): both compile. - spotless + scalastyle: clean. ## Roadmap Parts still to come: Rust driver-side planning (3a), Rust executor-side read path (3b), Scala claim/decline (4a), Scala execution — end-to-end native reads (4b), Change Data Feed (5), test battery + regression harness (6), and docs (7). Each is gated behind `-Pcontrib-delta`, so every intermediate state on `main` is safe for default builds. Tracking umbrella: #4366; part 1: #4700. --- 🤖 AI disclosure: this PR was prepared with assistance from [Claude Code](https://claude.com/claude-code) (Claude Opus 4.8), under the submitter's review and direction. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
