zhuqi-lucas commented on PR #13933: URL: https://github.com/apache/datafusion/pull/13933#issuecomment-2564651319
Hi @alamb @Dandandan , i submitted the first version PR for the automatically detect parquet file order and using the info to optimize for the plan. It's a very basic PR, we can add more follow-up issues to improve it. I have more questions and will try to create follow-up issues after this PR, for example: 1. Validation for the complex or invalid sort by cases, such as sort by ((a+1)*5 + b)? 2. We only using the first file sort metadata to expose to statistic, we need to consider if we need to handle for multi file cases, each one file may has different sort order? 3. Do we need to add more docs and use case for this new feature? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
