sam-1112 opened a new pull request, #5829: URL: https://github.com/apache/datafusion-comet/pull/5829
## Which issue does this PR close? Closes #5578. ## Rationale for this change `to_csv` never ran inside Comet by default. Every path through `CometStructsToCsv.getSupportLevel` returned `Unsupported` or `Incompatible`, and the serde did not mix in `CodegenDispatchFallback`. Any projection containing `to_csv` therefore fell back to Spark. The native implementation remains opt-in via `spark.comet.expression.StructsToCsv.allowIncompatible=true` and is not fully Spark-compatible. `to_json` is already Hybrid; `from_csv` and `schema_of_csv` already use JVM codegen dispatch. This PR gives `to_csv` the same default: Spark-generated `to_csv` runs through the JVM codegen dispatcher so the enclosing projection stays in the Comet pipeline. Spark 3.4/3.5 stringify nested array/map/struct fields with `Object.toString()`, so Spark's `ColumnarArray@…` cannot match the dispatcher's Arrow-backed `InputArray_col…` wrapper. Spark 4.0 ([SPARK-47497](https://issues.apache.org/jira/browse/SPARK-47497)) pretty-prints those nested values. Nested-result parity is therefore asserted only on Spark 4.0+. The production routing does not version-gate nested `to_csv`; the dispatcher still runs on 3.4/3.5, and the tests skip comparing those unstable strings. ## What changes are included in this PR? - Mix `CodegenDispatchFallback` into `CometStructsToCsv`. - Leave `getSupportLevel` unchanged: nested complex fields stay `Unsupported` on the native path; date/timestamp/binary and the remaining cases stay `Incompatible`. - Default (and nested) cases run Spark generated `to_csv` through the JVM codegen dispatcher on all supported Spark versions when the dispatcher is on. - `allowIncompatible=true` still selects the existing native path for inputs the native implementation accepts. - Docs: `to_csv` is listed as Hybrid rather than Native. This does not change Rust, JNI, protobuf, or native `to_csv` semantics. | Spark version | Input | Default path | Result vs Spark | | --- | --- | --- | --- | | All | Scalar fields | JVM codegen dispatcher | Match | | 3.4 / 3.5 | Nested array/map/struct fields | JVM codegen dispatcher | Not compared (`Object.toString()` vs Arrow wrapper) | | 4.0+ | Nested array/map/struct fields | JVM codegen dispatcher | Match (pretty strings) | | All | Dispatcher disabled | Spark fallback | Match | | All | `allowIncompatible=true` and native input is supported | Native opt-in | Routing verified | ## How are these changes tested? - `CometCsvExpressionSuite`: default scalar dispatcher (`assertCodegenRan`), NULL struct, dispatcher-disabled fallback, native opt-in (`totalLookups == 0`). Nested array parity is `assume(isSpark40Plus)`. - SQL: `csv.sql` (scalar, all versions), `csv_fallback.sql` (dispatcher off), `to_csv_nested.sql` with `-- MinSparkVersion: 4.0`. - Spark 3.4 / 3.5 / 4.0 / 4.1 targeted CSV suites; nested cases canceled/skipped on 3.4/3.5. `CometJsonExpressionSuite` / `CometJsonJvmSuite` confirm `to_json` routing is unchanged. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
