adriangb commented on issue #21207: URL: https://github.com/apache/datafusion/issues/21207#issuecomment-5026846741
The one note w.r.t. a single global membership: the merging itself can be expensive if it implies copying 128 hash tables into 1 hash table (for a plan with 128 partitions). For the smaller `InList` expressions I think the cost can more easily be amortized / we are already copying the data. We tried options that don't involve copying, like hashing once and probing each partition's hash table without rehashing, but that proved to be slower at high partition counts, see https://github.com/apache/datafusion/pull/21931. cc @zhuqi-lucas since you're working on this stuff too -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
