andygrove opened a new issue, #7203:
URL: https://github.com/apache/datafusion-comet/issues/7203
### Describe the bug
#4727 made `collect_list` and `collect_set` native in aggregate chains with
a `PartialMerge`. This is the plan Spark builds when a query mixes
`count(DISTINCT ...)` with other aggregates. 1.1.0 kept any such chain in Spark
(`hasNativeArrayBufferAgg` in `CometExecRule`).
On Spark versions before 4.2, with
`spark.comet.exec.strictFloatingPoint=false` (the default), native
`collect_set` keeps -0.0 and 0.0 as two entries and merges every NaN into one.
Spark does the opposite: it collapses the zeros and keeps each NaN. The
compatibility guide already documents this for plain `collect_set`, which was
native in 1.1.0 too. What is new in 1.2.0 is that the distinct shape now
reaches it. On 1.1.0 that shape matched Spark.
### Steps to reproduce
Spark 4.1, default configs:
```sql
SELECT x, count(DISTINCT y), sort_array(collect_set(z))
FROM VALUES (1, 1, CAST('NaN' AS DOUBLE)), (1, 2, CAST('NaN' AS DOUBLE)),
(1, 3, -0.0D), (1, 4, 0.0D), (1, 5, 1.0D) AS t(x, y, z)
GROUP BY x
```
- Spark: `[0.0, 1.0, NaN, NaN]` (or `[-0.0, 1.0, NaN, NaN]` when the rows
come from Parquet)
- 1.2.0: `[-0.0, 0.0, 1.0, NaN]`
- 1.1.0: matches Spark, because the aggregate ran in Spark
FLOAT behaves the same way. The divergence shows up both when the partial
aggregate runs in Spark and when it runs natively over a Parquet scan.
### Expected behavior
The distinct shape matches Spark, as in 1.1.0. Two ways to get there:
- Fall back for `collect_set` over FLOAT or DOUBLE elements in a
`PartialMerge` chain on Spark before 4.2.
- Extend the input normalization from #5312 to Spark 3.4 through 4.1.
### Additional context
This is a regression since 1.1.0, found in the branch-1.2 audit. Related:
#5312 and #6385. Every other element type checked matched Spark through both
the Spark-partial decoder and the native path. That covered decimals up to
(38,10), binary, temporal types, and nested arrays, structs and maps.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
