jayzhan211 commented on PR #25126:
URL: https://github.com/apache/datafusion/pull/25126#issuecomment-5642664266
> > Wouldn't this work for max as well?
>
> It looks like @neilconway predicted this one: #22644.
>
> It sounds like what we need is
>
> #### A trait method on `AggregateUDFImpl::distinct_handling() ->
DistinctHandling`
> ```rust
> /// How an aggregate function treats the `DISTINCT` modifier.
> ///
> /// Mathematically, `Ignored` means the function's merge operation is
> /// idempotent (its state forms a semilattice): f(S ⊎ S) = f(S), so
> /// removing duplicates from the input cannot change the result.
> #[derive(Debug, Clone, Copy, PartialEq, Eq)]
> pub enum DistinctHandling {
> /// The result is the same with or without `DISTINCT`, so the planner
> /// is free to drop it. `min`, `max`, `bool_and`, `bit_or`, ...
> Ignored,
> /// The accumulator honours `AccumulatorArgs::is_distinct` and
> /// deduplicates its input. `count`, `sum`, `avg`, `array_agg`, ...
> Honored,
> /// The accumulator does not implement `DISTINCT`. Planning
> /// `f(DISTINCT ...)` is an error. `corr`, `regr_*`, `nth_value`, ...
> Unsupported,
> }
>
> pub trait AggregateUDFImpl {
> /// How this function treats the `DISTINCT` modifier.
> ///
> /// Return [`DistinctHandling::Ignored`] for duplicate-insensitive
> /// functions so that `f(DISTINCT x)` is planned as `f(x)`, and
> /// [`DistinctHandling::Unsupported`] if the accumulator does not
> /// read `is_distinct`, so that `DISTINCT` is rejected at planning
> /// time rather than silently ignored.
> fn distinct_handling(&self) -> DistinctHandling {
> DistinctHandling::Honored
> }
> }
> ```
>
> We would set this to `Unsupported` for Correlation, Covariance, Regr,
ApproxPercentileCont, NthValue, Stddev, Variance and ApproxMedian.
>
> #### An optimizer rule that runs before `SingleDistinctToGroupBy` to
remove `DISTINCT` where it wasn't needed so that `SingleDistinctToGroupBy`
doesn't fire
+1 for this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
