andygrove commented on PR #6449: URL: https://github.com/apache/datafusion-comet/pull/6449#issuecomment-5915408998
This also fixes #6464, which the 1.1.0 regression audit (#6399) found in rc1. Since #5362 and #5667, the native explode slices its output instead of gathering it, so exploding an array of structs with a boolean field hands the JVM sliced boolean children, and they come back wrong after the first `spark.comet.batchSize` rows of each input batch. I ran the reproducer from #6464 on this branch, along with the `posexplode`, `explode_outer`, `named_struct` and Scala UDF variants. All of them match Spark in 3 runs out of 3, and all of them fail on rc1. So this backport covers #6464 as well as #6424. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
