alamb commented on PR #25988: URL: https://github.com/apache/datafusion/pull/25988#issuecomment-6721525769
> The native plan is now opt-in. The default keeps the existing two-phase rewrite. Under a 1 MiB memory limit, the reviewer's query with two groups and 100,000 distinct values completes with the rewrite and returns 2, 100000; the native plan runs out of memory. Native DISTINCT must fit each group's distinct values in memory, including when merging spilled or partial states. Do we have a plan for making this not opt-in (turning it on by default)? Given it isn't on by default, I would say it hasn't actually closed https://github.com/apache/datafusion/issues/25936 has it? It seems to me like this will just be technical debt (rarely used as it is behind a flag) unless we work to get it on by default -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
