geoffreyclaude commented on PR #23014: URL: https://github.com/apache/datafusion/pull/23014#issuecomment-5056264840
> One way we could potentially reduce the code size would be to minimize the parts of the code that really need specializations. @alamb I agree, and I pushed a new remediation commit along those lines. The `IN`-list values are now stored at runtime, and the filter chooses a comparison function for that list length when it is built. That function is called once per input array, while the rest of the filter is shared. I didn't use `chunks_exact` exactly: keeping one small comparison function per list length was the simplest option that preserved performance. Looking at the release artifacts, the number of copies of `BranchlessFilter::try_new` and `contains` drops from 507 to 27. About 86 small comparison functions remain. I also rebuilt release `datafusion-cli` for the previous and current versions on the same base, toolchain and arm64 machine: - executable: 82,316,496 → 77,773,840 bytes (-5.5%) - Mach-O `__TEXT`: 78,479,360 → 73,990,144 bytes (-5.7%) - Mach-O `__text`: 65,119,604 → 60,957,044 bytes (-6.4%) That brings the release binary back to roughly the 74M `main` baseline from your measurement. For performance, I compared the two versions across 28 focused Criterion cases. The worst median regression was 2.45%; the next two were 1.58% and 1.57%. None reached 3%, and many cases were flat or faster. Thanks for pushing on this! I think this follow-up addresses the main concern without giving up the performance gains. WDYT? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
