LiaCastaneda commented on PR #25185:
URL: https://github.com/apache/datafusion/pull/25185#issuecomment-5637165203
dictionary value arrays are commonly re-used in prod workloads, this
inneficiency is something we noticed in our service, we saw the following
scenario:
```
SELECT user_id, count(*)
FROM events
GROUP BY user_id
```
with `user_id` with type Dictionary<_, Utf8> and with 28.5M rows, ~28.5M
distinct we get the following:
```
AggregateExec: mode=FinalPartitioned, gby=[user_id] ← (4)
RepartitionExec: Hash([user_id], 12) ← (3)
AggregateExec: mode=Partial, gby=[user_id] ← (2)
DataSourceExec ← (1)
```
becuase there is no group by the partial aggregate goes through
`vectorized_append`, it hashes all of the values of the batch (which is
expensive because the column is high cardinality -> 28.5M hashes). Then, in the
RepartitionExec the values are inherited (Arc cloned) and then the final
aggregation gets the same values, however since it goes thorugh
`vectorized_append`, in the current code, it will hash again all the values
(25M) of the batch even if its unnecessary becuase its the same array
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
