andygrove opened a new pull request, #7281: URL: https://github.com/apache/datafusion-comet/pull/7281
## Which issue does this PR close? Backport of #6940, which closes #6929. ## Rationale for this change Native Parquet writes cannot run where S3A's magic committer commits the write, and Spark 4.1 makes it the default whenever spark-hadoop-cloud is on the classpath (SPARK-47618). #6940 lets the native writer produce such a write by having the JVM create each file through S3A and the native writer supply only its bytes. This is a feature rather than a bug fix. `branch-1.2` has not shipped its `.0` release, so it is still taking what `main` gains after the cut (see [Which Branches Take Backports](https://github.com/apache/datafusion-comet/blob/main/docs/source/contributor-guide/backporting.md#which-branches-take-backports)), and the release manager decides whether it stays. It is not going to `branch-1.1` or `branch-1.0`, which take regression fixes only. There is no newer release branch. ## What changes are included in this PR? The cherry-pick of f1121ca146 (`git cherry-pick -x`), with one adaptation: - `native/core/src/execution/jni_api.rs`: `Native.setOutputStream` calls `get_execution_context(exec_context)` without the `?`. On `main` that function returns a `CometResult` (it rejects a null handle with an `Internal` error), but on `branch-1.2` it returns the `&mut ExecutionContext` directly. `setOutputStream` already rejects a zero handle with a `NullPointer` error before the call, so no check is lost. The change that made the function fallible is not backported. Everything else is identical to the source commit. ## How are these changes tested? Built on `branch-1.2` with the cherry-pick, because a clean pick can still depend on code that landed after the cut: - `cargo check --all-targets --workspace`, `cargo clippy --all-targets --workspace -- -D warnings` and `cargo fmt --all --check` in `native/`. - `./mvnw test-compile` on the default profile (Spark 4.1) and on `-Pspark-3.5`. - The Rust tests for the stream writer, the sink and the planner: 142 passed in `datafusion-comet`. `CometParquetWriterSuite`, `CometParquetWriterRetrySuite` and the other suites the source PR changed run in CI, which runs every tier on a pull request against a release branch. `ParquetWriteWithMagicCommitterSuite` needs MinIO and is manual, as on `main`; I have not run it on this branch. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
