andygrove commented on code in PR #7543:
URL: https://github.com/apache/datafusion-comet/pull/7543#discussion_r4588207121
##########
spark/src/test/scala/org/apache/comet/exec/CometAggregateSuite.scala:
##########
@@ -3431,6 +3431,59 @@ class CometAggregateSuite extends CometTestBase with
AdaptiveSparkPlanHelper {
}
}
+ test("skip partial aggregation converts AVG rows to partial states") {
+ withTempDir { dir =>
+ val path = new Path(dir.toURI.toString, "input").toUri.toString
+ withSQLConf(CometConf.COMET_ENABLED.key -> "false") {
+ // Mostly distinct keys within each task, with some keys repeating
across tasks so the
+ // final aggregate merges converted states with each other. Values
include nulls.
+ spark
+ .range(0L, 16384L, 1L, 8)
+ .selectExpr(
+ "id % 12000 AS k",
+ "CASE WHEN id % 5 = 0 THEN NULL ELSE CAST(id % 17 AS INT) END AS
i",
+ "CASE WHEN id % 7 = 0 THEN NULL ELSE CAST(id % 11 AS DOUBLE) / 3
END AS d",
+ "CASE WHEN id % 3 = 0 THEN NULL ELSE CAST(id % 13 AS DECIMAL(10,
2)) / 7 END AS small",
Review Comment:
Right, `/ 7` made it `DECIMAL(14, 6)`, so it never took the `UnscaledValue`
path. Dropped the division, so `small` is now `DECIMAL(10, 2)`, and
`avg(small)` is asserted to skip. It passes (8ec026f4f).
##########
docs/source/user-guide/latest/tuning/operators.md:
##########
@@ -172,7 +172,7 @@ pruning does not require that setting.
Set `spark.comet.exec.aggregate.skipPartial.enabled=true` to let Comet bypass
partial hash
aggregation for high-cardinality grouping when it is not reducing the number
of rows enough. This
experimental optimization is disabled by default. It currently applies only to
fused native
-shuffle-writer plans whose partial aggregates are grouping-only or
single-argument `COUNT`.
+shuffle-writer plans whose partial aggregates are grouping-only,
single-argument `COUNT`, or `AVG` over non-decimal inputs.
Review Comment:
Agreed, the gate is on the `DOUBLE` result. Both lines and the `CometConf`
doc now say `AVG` over any input except a decimal with precision above 11. The
Eligibility paragraph also says that Spark rewrites the small-decimal case into
a `DOUBLE` `AVG`. Rewrapped to the section's width.
##########
native/spark-expr/src/agg_funcs/avg.rs:
##########
@@ -334,12 +334,33 @@ where
])
}
+ /// Each row's own partial state, as `state` emits it: the value and a
count of one, or a
+ /// zero sum and count for a null or filtered-out row.
fn convert_to_state(
&self,
- _values: &[ArrayRef],
- _opt_filter: Option<&arrow::array::BooleanArray>,
+ values: &[ArrayRef],
+ opt_filter: Option<&arrow::array::BooleanArray>,
) -> Result<Vec<ArrayRef>> {
- not_impl_err!("Input batch conversion to state not implemented")
+ let values = values[0].as_primitive::<T>();
+ let mut sums = Vec::with_capacity(values.len());
+ let mut counts = Vec::with_capacity(values.len());
+ for idx in 0..values.len() {
Review Comment:
Done. With no nulls and no filter, the input values buffer is reused as the
sums and the counts are all ones. Otherwise one mask comes from
`NullBuffer::union` of `filter_to_nulls(filter)` and the input nulls, and one
zip builds sums and counts. Masked rows keep a zero sum for `merge_batch`.
`filter_to_nulls` is now shared from `collect.rs` instead of copied.
##########
native/spark-expr/src/agg_funcs/avg.rs:
##########
@@ -365,4 +386,32 @@ mod tests {
assert_eq!(partial.evaluate()?, ScalarValue::Float64(None));
Ok(())
}
+
+ /// Converting rows straight to partial states, as a partial aggregate
that stops aggregating
+ /// does, gives each row the state that aggregating it alone into a group
would.
+ #[test]
+ fn convert_to_state_matches_one_row_groups() -> Result<()> {
+ use arrow::array::BooleanArray;
+ let values: ArrayRef = Arc::new(Float64Array::from(vec![
+ Some(1.5),
+ None,
+ Some(-2.0),
+ Some(4.0),
+ ]));
+ let filter = BooleanArray::from(vec![Some(true), Some(true),
Some(true), Some(false)]);
Review Comment:
Added. The filter has a `None` slot, and the test runs inputs with and
without nulls, with and without a filter, plus sliced inputs and filters
(`values.slice(1, 3)` and sliced filters at offsets 1 and 2). Each case is
compared against one-row-group `update_batch` + `state`. The Scala test now
also asserts that `avg(i) FILTER (WHERE d > 1)` skips and matches Spark, with
`d` null on some rows.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]