2010YOUY01 commented on PR #23777: URL: https://github.com/apache/datafusion/pull/23777#issuecomment-5288633670
> Instead of a config option, could we use a flag that gets set by select operators, such as hash aggregate? Because for hash aggregate we already know that sliced RecordBatches will be produced from a huge initial RecordBatch. Yes, I think starting with only aggregate is a good idea > With the config option my concern is that we'd be trading correctness for performance, since without the deduplication the hash aggregation shows heavily inflated output_bytes metrics. I agree. That also adds extra complexity. I imagine the performance impact would be minimal, since a hash set whose size is proportional to the number of batches is much smaller than the actual workload. We just need to verify that in https://github.com/apache/datafusion/issues/24310 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
