alamb commented on PR #15466: URL: https://github.com/apache/datafusion/pull/15466#issuecomment-2792347897
I tried briefly to reproduce the performance improvements reported above and it seems like I can: [q14.sql.txt](https://github.com/user-attachments/files/19683249/q14.sql.txt) ```shell andrewlamb@Andrews-MacBook-Pro-2:~/Downloads$ for i in `seq 1 5`; do datafusion-cli -f q14.sql ; done | grep seconds Elapsed 0.374 seconds. Elapsed 0.386 seconds. Elapsed 0.378 seconds. Elapsed 0.366 seconds. Elapsed 0.366 seconds. ``` ``` andrewlamb@Andrews-MacBook-Pro-2:~/Downloads$ for i in `seq 1 5`; do ./datafusion-cli-alamb_test_upgrade_54 -f q14.sql ; done | grep seconds Elapsed 0.384 seconds. Elapsed 0.368 seconds. Elapsed 0.356 seconds. Elapsed 0.373 seconds. Elapsed 0.377 seconds. ``` I poked around and I can't quite figure out if the improvement is related to concat batches or maybe reducing the number of IOs for reading the parquet metadata 🤔 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
