ErikBPF opened a new pull request, #5903:
URL: https://github.com/apache/datafusion-comet/pull/5903
## Which issue does this PR close?
Closes #5239.
## Rationale for this change
Grouped collection aggregation over structs with required fields can fail
when final aggregation spills. Native collection state normalizes nested
fields to nullable, but Comet's intermediate output schema retains the
original required fields. The exchange casts the state back to that strict
schema, and final aggregation subsequently rejects nullable state emitted
for spilling.
## What changes are included in this PR?
Use Spark's existing `asNullable` conversion for the element type of both
`CollectList` and `CollectSet` intermediate buffers. Add one regression
covering both functions, with required fields created after the Parquet
scan, Spark result comparison, native execution, and a positive aggregate
spill assertion.
This fixes Comet's intermediate schema declaration independently of
apache/datafusion#24767, which changes upstream Spark accumulator typing.
## How are these changes tested?
The unchanged regression failed on base
`f69c4c81b9429e327ea95658530ae4ed4ed19635` with a strict-versus-nullable
struct schema mismatch during final aggregation, then passed with this
change using the release native library and Spark 4.1.3.
```bash
./mvnw -B -Prelease test -Dtest=none \
'-Dsuites=org.apache.comet.exec.CometAggregateSuite collect_list and
collect_set over non-nullable nested fields survive spilling'
```
The complete `CometAggregateSuite` also passed: 99 succeeded, zero failed,
and the two pre-existing metric tests remained ignored. The full Maven
reactor, including Scalastyle and Spotless checks, succeeded.
The regression also passed with Spark 3.5.9 / Scala 2.12.18 after a clean
full-reactor build using `-Prelease -Pspark-3.5`, including style and
formatting checks. Both Spark versions loaded the same release native
library. The full aggregate suite was run on Spark 4.1.3 only.
The first full-suite attempt hit the test unit's 1,024-task limit while
Spark created an AQE thread. Raising that task limit to 4,096 allowed the
suite to finish; the CPU and 32 GiB memory caps were unchanged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
