yashrb24 opened a new issue, #25033: URL: https://github.com/apache/datafusion/issues/25033
### Is your feature request related to a problem or challenge? Today we evaluate all sort expressions before sorting. Some of this work may be unnecessary when earlier keys already determine the order. For example: ```sql SELECT id FROM events ORDER BY score DESC, md5(payload) LIMIT 10; ``` If score is enough to order two rows, comparing them does not require md5(payload). For TopK, rows whose score cannot reach the result do not need the second key either. ### Describe the solution you'd like Evaluate sort keys in stages, possibly a group of keys at a time: - Sort by the first group of keys. - Evaluate the next keys only for rows tied on the earlier keys. - Repeat until the order is resolved. This could help general sorting. With LIMIT, we could also skip later keys for rows already excluded from the result. The problem here is that the extra sorting and filtering steps could outweigh the savings for cheap keys or many ties so grouping/chunking the evaluation of keys should be helpful. Since it would involve some considerable change to the sort mechanics, wanted some inputs before I draft out changes for this, in case it has been considered before. I could not find any previous issue/PR for this ### Describe alternatives you've considered _No response_ ### Additional context https://github.com/apache/datafusion/pull/22603 had explored skipping sort-key work for fully rejected TopK batches. My proposed change here is broader in scope. Inspiration is from Spark, where it evaluates expressions inside its sort comparator and stops when an earlier key differs. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
