Rich-T-kid commented on PR #26816: URL: https://github.com/apache/datafusion/pull/26816#issuecomment-6452428138
> It looks to me that you are working the same angle as @jayzhan211 in https://github.com/apache/datafusion/pull/26703 (Q18/Q33/Q34 ~ 1.8–1.9×, total −28 % vs −29 % here), maybe you two could join forces on this one? I didn't have time to go through all the details of both PRs, but if you check and still have doubts, one quick way to check is to rebase this on https://github.com/apache/datafusion/pull/26703 and re-run clickbench_partitioned, if the improvements stack (at least partially), then the two PRs don't match exactly, otherwise I suggest to collaborate on this. WDYT? from my reading of the PR it tries to solve the same problem in a different way. Ill review that PR and share my thoughts there but I think we should go with re-using hashes instead of limiting the amount of hash work that we do to a per-row cap. >Also, both runs compare against a baseline with enable_rle_to_dictionary=true, which is off by default. It would help to also show the branch against the default config, so we know how far dictionary reads still are from plain strings after this change. Yes, this is intentional. Initially, some benchmarks were 5x slower than main, which made it hard to see what was moving the needle in a positive direction when comparing directly against main. The end goal is to be faster than main, which uses utf8Views, but there’s still a decent chunk of work to go. So as I iterate, I’m comparing my dictionary encoding optimizations against dictionary encoding on main. Once the benchmark results are closer, I’ll compare against main directly. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
