2010YOUY01 commented on PR #16889: URL: https://github.com/apache/datafusion/pull/16889#issuecomment-3126813407
I tried to eliminate redundant batch transformations, the speedup goes from 2X -> 3X! Through the flamegraph, I can see that over 50% of the time is now spent evaluating the join filter (e.g. `(c1 + c2) % 10 = 1`). It seems there may not be any more low-hanging fruits for further speed improvements. After a few more chores, this should be ready for review: - [ ] figure out the output ordering properties and update test - [ ] Metrics accounting - [ ] Polish(more comments and clean-up) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
