gruuya commented on PR #25292: URL: https://github.com/apache/datafusion/pull/25292#issuecomment-5758881959
Ok, I get it now, the bucket bitmap approach you suggest does seem like a more sophisticated approach @jayzhan211. We trade-off pruning precision against pruning speed: make it fixed cost but we sometimes scan more false positives than we need to. The pathological case in that regards seems to be when there are a lot of (file/row-group/page) containers, and the build side has few-ish values spanning a big range. In that case each bucket spans a large-ish sub-range as well, and the ones that are populated by the sparse values will falsely "light-up" many redundant containers alongside the correct one. Probably mitigated easily to some degree by just defaulting to 2^20 for the bucket size. Either way it would be a net win over the default state today. Let me see how much of the PR is salvageable if we pick that direction. Doesn't make much sense to me to merge this with the new config and then remove it soon after in the follow-up. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
