andygrove commented on code in PR #6473:
URL: https://github.com/apache/datafusion-comet/pull/6473#discussion_r4150045926
##########
spark/src/test/scala/org/apache/comet/exec/CometGenerateExecSuite.scala:
##########
@@ -659,4 +659,67 @@ class CometGenerateExecSuite extends CometTestBase {
}
}
+ // The native explode slices each output batch out of the exploded child
instead of gathering
+ // it, so an exploded boolean, or a boolean field of an exploded struct,
leaves native at a
+ // non-zero bit offset. Arrow Java ignores that offset on import, so native
has to zero it at
+ // every level before export, including in a struct built over the booleans
and in the input to
+ // a Scala UDF, or they come back wrong after the first output batch.
+ // https://github.com/apache/datafusion-comet/issues/6464
+ private def withBooleanArrays(numRows: Int, arrayLength: Int)(f: => Unit):
Unit = {
+ withTempPath { dir =>
+ // One file, so a single input batch explodes into several output
batches.
+ withSQLConf(CometConf.COMET_ENABLED.key -> "false") {
+ spark
+ .range(0, numRows, 1, 1)
Review Comment:
Done in 310a7eacf9. The fixture now takes `Long` arguments and calls
`spark.range(0L, numRows, 1L, 1)`.
The full reactor Spark 3.5/JDK 17 strict-warning compilation passes locally:
`./mvnw -o -B test-compile -Pspark-3.5 -Pstrict-warnings -DskipTests`. CI is
still running.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
