rishvin opened a new issue, #16336: URL: https://github.com/apache/datafusion/issues/16336
### Describe the bug This ticket is related to [#1820](https://github.com/apache/datafusion-comet/issues/1820) from Comet. We are working on using Datafusion's Sha2 (`SparkSha2`) implementation in Comet. However when making changes, found 2 issues (See [here](https://github.com/apache/datafusion-comet/issues/1820#issuecomment-2953616585) for full details), - `SparkSha2` is not fully compliant with the Apache Spark Sha2 response. The Apache Spark returns Sha2 hex output in lowercase however `SparkSha2` returns the response in uppercase. Eg, ``` Spark Sha2 response - 2C83E9E8A39D60F7FCD3CFEC29C154260AA069F91CD40C972756F9354C64594E Datafusion SparkSha2 response - 2c83e9e8a39d60f7fcd3cfec29c154260aa069f91cd40c972756f9354c64594e ``` - `SparkSha2` pattern matching expects Sha2 [bit-length type](https://github.com/apache/datafusion/blob/1daa5ed5cc51546904d45e23cc148601d973942a/datafusion/spark/src/function/hash/sha2.rs#L141) to be `Uint32`, however Spark (and Comet) doesn't support `Uint32` type. This is due to the limitation of JVM which only support signed types. Because of which Comet is not able to send `UInt32` type for bit-length and hence pattern matching fails. This ticket is about making two fixes, - Make `SparkSha2` response compliant with Spark. - Add support for `Int32` ### To Reproduce _No response_ ### Expected behavior _No response_ ### Additional context _No response_ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
