martin-g opened a new pull request, #25404: URL: https://github.com/apache/datafusion/pull/25404
## Which issue does this PR close? - Closes #5562 ## Rationale for this change Currently `TableProvider` provides the `limit` argument to the `scan()` method to request a maximum number of rows. There is no way to tell the implementation to skip some of the rows, for example to fetch the second/third/Nth page of rows (i.e. SQL `... LIMIT 20 OFFSET 40`). Adding an additional field to ScanArgs (named `offset` or `skip`) will make it possible for implementations to override the `scan_with_args()` method and optimize their scan to read and return only the requested rows. ## What changes are included in this PR? * A new field named `offset` is added to `ScanArgs`, with a setter and a getter. * A new method is added to the `TableProvider` trait - `supports_offset_pushdown() -> bool`. By default it returns `false` but any implementation that can support skipping of rows could override it to return `true` and combined with a custom implementation of `scan_with_args()` to optimise its data scan/read. * Update some callers of `TableProvider::scan()` to use `::scan_with_args()` where they could support offset push down * Update the migration guide for 56.0.0 with a section about the offset pushdown support Note: `datafusion-ffi` is **not** updated because it does not expose `scan_with_args()` yet. ## What is the testing strategy for this PR? New unit tests are added for the implementations which support offset pushdown. ## Are there any user-facing changes? The new functionality is opt-in! All currently existing custom implementations of `TableProvider` trait will continue to compile and run without any modifications. Any custom implementation that wants to make use of the new functionality will need to override `TableProvider::supports_offset_pushdown()` to return `true` and make use of `ScanArgs::offset` in its `scan_with_args()` implementation. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
