andygrove opened a new pull request, #5140: URL: https://github.com/apache/datafusion-comet/pull/5140
## Which issue does this PR close? Cherry-pick of #5079 (merged to `main` as `76b9d2eee`) onto `branch-1.0`. ## Rationale for this change `CometCast` was the last serde on the incompatible/unsupported path that had no codegen dispatch. Without it, a single cast that Comet declines to run natively forces the whole projection back to Spark, so the surrounding native operators are lost too. The cases affected are all reachable with default or common configuration: - `spark.sql.legacy.castComplexTypesToString.enabled=true` (struct/array/map to string) was `Unsupported`, forcing full fallback. - Non-`CORRECTED` `spark.sql.legacy.timeParserPolicy` (`LEGACY`, `EXCEPTION`) on string to date/timestamp casts. - `MapType` to string, and struct/array containing a `MapType`, which have no native implementation. - Pre-existing `Incompatible` cases (negative-scale decimal to string, float/double to decimal rounding). `branch-1.0` already carries `CodegenDispatchFallback` and the serdes that use it, so this is the same mechanism already shipping for `In`/`InSet`, strings, datetime, arrays, and maps — `CometCast` was simply missed. Getting it into 1.0.0 avoids a release where a legacy cast config silently costs users native execution of an entire projection. ## What changes are included in this PR? A `git cherry-pick -x` of `76b9d2eee`. The content is identical to the commit on `main`; the two `dev/diffs/*.diff` hunks auto-merged (only surrounding `comet.version` context differs on this branch, since `branch-1.0` is at `1.0.0` rather than `1.1.0-SNAPSHOT`), and no adaptation was needed. - `CometCast` mixes in `CodegenDispatchFallback`, so casts it cannot match Spark bit-for-bit run Spark's own `doGenCode` inside the Comet pipeline instead of falling the projection back. - `spark.sql.legacy.castComplexTypesToString.enabled=true` now routes through the dispatcher; the complex-type check is deduplicated against `DataTypeSupport.isComplexType`, and the `SQLConf` lookup is reordered to run only when the source type is complex. - `VariantType` (Spark 4, SPARK-45827) is rejected in `CometCast.getSupportLevel` / `isSupported` via a new `isVariantType` in `CometTypeShim` (stubbed `false` on Spark 3.x). Neither the native serde nor the codegen dispatcher can carry `VariantType` in the data args or return type, so these casts must reach Spark. The guard also covers the `Literal` short-circuit branch, where constant folding would otherwise produce a `Literal[VariantType]` that no downstream serde can serialize. - Tests: `expect_fallback` markers dropped from the non-legacy `cast_complex_types_to_string` cases that now stay in the Comet pipeline, plus a missing `query` directive restored above the array-of-map case (it was previously an orphan `SELECT` that never ran); `CometCastSuite` checks the Comet operator rather than a Spark fallback reason for `ArrayType(DateType)` to an unsupported `ArrayType`; `CometExpressionSuite`'s `explain comet` expectations updated for the `CalendarIntervalType` to string cast, which no longer emits a fallback reason. - `dev/diffs/4.0.2.diff` and `4.1.2.diff`: loosen `VariantEndToEndSuite`'s codegen-mode plan-type assertion to also accept `CometNativeColumnarToRowExec`, since under Comet the plan is wrapped rather than having `WholeStageCodegenExec` at the top. ## How are these changes tested? By the tests carried in the cherry-pick. Verified locally on this branch with a debug native build: - `CometSqlFileTestSuite` filtered to `cast_complex_types_to_string` — 3 tests pass (legacy file, plus non-legacy with dictionary encoding on and off). - `CometCastSuite` — 163 tests pass on the default profile, and 162 pass (1 canceled) under `-Pspark-3.5`, which exercises the `spark-3.x` `isVariantType` stub. - `CometExpressionSuite` — 138 tests pass, covering the updated `explain comet` expectations. - `./mvnw spotless:check scalastyle:check` clean on `comet-spark`. - Both updated Spark diffs verified with `git apply --check` against `v4.0.2` and `v4.1.2` checkouts. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
