discord9 opened a new pull request, #25068: URL: https://github.com/apache/datafusion/pull/25068
## Which issue does this PR close? No linked issue. Independently reproduced physical-optimizer correctness fix. ## Rationale for this change Repartitioning a memory source with an existing fetch rebuilds the source without that fetch. A one-partition scan containing batches `[0,1]` and `[2,3]` with fetch 1 returns `[0]` before optimization, but all four rows after source repartitioning. This also occurs through EnsureRequirements and one/two complete default physical-optimizer passes with a parent filter and no outer limit. The fetch is enforced separately for each original partition. Copying it onto newly formed partitions would still change the result, rather than preserve the original cap. This is demonstrated for directly constructed physical plans and their optimization; an ordinary SQL-planning reproduction is not established. ## What changes are included in this PR? Have `MemorySourceConfig` decline source-level repartitioning whenever a fetch is present, before either sorted or unsorted repartitioning. The optimizer can still insert an ordinary `RepartitionExec` over the existing capped source, distributing rows already admitted by the original caps. This uses the existing optional repartition contract, with no changes to optimizer configuration, statistics, or shared fetch accounting. It is independent of #23800 and #25065. ## What is the testing strategy for this PR? - Uncapped source repartition remains enabled and emits all rows. - Eligible two-batch sources with fetch 0 decline repartition and remain empty. - Two original capped partitions preserve their prefix values `[0,10]`. - An eligible projected/sorted source preserves its schema, ordering, and capped value `[10]`. - EnsureRequirements and one/two full optimizer passes retain `[0]` and use ordinary round-robin repartition over the original one-partition fetched source. Removing only the guard makes the independently eligible projection/ordering regression fail. The guard was restored byte-exactly; the original direct/API and optimizer reproductions also demonstrated the pre-fix one-row-to-four-row change. Verified locally: - `cargo check -p datafusion-datasource` - `cargo test -p datafusion-datasource memory:: --lib` — 15 passed - `cargo test -p datafusion --test core_integration physical_optimizer::enforce_distribution::` — 88 passed - `cargo fmt --all --check` - `cargo clippy --workspace --all-targets --all-features -- -D warnings` The full workspace runtime test suite was not run. ## Are there any user-facing changes? Preserves the results of already-capped memory scans during physical optimization. No public API or configuration changes. Source-level repartition is declined for capped sources; ordinary downstream repartition remains available. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
