sam-1112 opened a new pull request, #5835:
URL: https://github.com/apache/datafusion-comet/pull/5835
## Which issue does this PR close?
Closes #5581.
## Rationale for this change
`hash` and `xxhash64` currently report `Unsupported` for `DecimalType`
inputs with precision greater than 18, including decimals nested inside arrays,
structs, and maps. `sha2` also reports `Unsupported` when `numBits` is not
foldable.
Because the corresponding serdes do not mix in `CodegenDispatchFallback`,
these cases cause the enclosing projection to fall back to Spark. This matters
for bucketing, partitioning, and deduplication workloads where retaining the
surrounding operator in Comet is valuable.
Spark already implements the required behavior in generated code:
`HashExpression.doGenCode` hashes wide decimals using Java `BigDecimal`, while
`Sha2.doGenCode` supports a non-literal `numBits`. The JVM codegen dispatcher
can execute that code inside the Comet pipeline without requiring changes to
the native hash implementations.
`TimeType` handling is intentionally unchanged.
## What changes are included in this PR?
- Mix `CodegenDispatchFallback` into `CometMurmur3Hash`, `CometXxHash64`,
`CometSha1`, and `CometSha2`.
- Leave `HashUtils.unsupportedReasonFor` and the native hash kernels
unchanged.
- Supported inputs continue to use the native path.
- Unsupported but dispatchable cases now run Spark's `doGenCode` through
the JVM codegen dispatcher.
- Preserve the existing Spark fallback when the dispatcher is disabled. The
fallback reason identifies the expression and includes
`spark.comet.exec.scalaUDF.codegen.enabled=false`.
- Update `hash_funcs.md`.
- Mark `hash`, `xxhash64`, `sha`, `sha1`, and `sha2` as Hybrid in
`expressions.md`.
## How are these changes tested?
Extended `CometHashExpressionSuite` with coverage for:
- Wide-decimal `hash` and `xxhash64`:
- `DECIMAL(20, 2)` and `DECIMAL(38, 10)`.
- Decimals nested inside arrays, structs, and maps.
- Spark result parity, retention of the Comet operator, and verification
through `assertCodegenRan`.
- Native-path controls:
- Hashing decimals with precision less than or equal to 18.
- `sha2` with a literal `numBits`.
- Both retain zero dispatcher lookups.
- Non-foldable `sha2(payload, num_bits)`:
- Valid and invalid `numBits` values.
- Null payloads and null `numBits`.
- Verification that the dispatcher executes and results match Spark.
- Dispatcher-disabled behavior:
- Wide-decimal hashing and non-foldable `sha2` safely fall back to Spark.
- Fallback reasons contain the expected expression name and configuration
key.
The newly added routing tests in `CometHashExpressionSuite` passed on Spark
3.5 and Spark 4.1.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
