Ruchirtripathi opened a new pull request, #27554:
URL: https://github.com/apache/datafusion/pull/27554

   ## Which issue does this PR close?
   
   - Closes #27472
   
   ## Rationale for this change
   
   In PR #27393, COPY semantics were tightened to only preserve sort order when 
the query specifies an explicit `ORDER BY`. However, it did so by checking 
`input_exec.downcast_ref::<SortExec>()`.
   When inner plans are complex—such as when `LogicalPlanBuilder` wraps the 
plan in a `ProjectionExec` to project out unselected sort columns (e.g., 
`SELECT id FROM (...) ORDER BY id, ts`), or when `LimitExec` / 
`CoalescePartitionsExec` are present—`downcast_ref::<SortExec>()` returns 
`None`. Consequently, `DataSinkExec` does not recognize the explicit ordering 
requirement, losing the sort order.
   
   ## What changes are included in this PR?
   
   1. Added `has_explicit_sort` to verify whether the logical plan specifies an 
explicit top-level sort (traversing through enclosing `Projection`, `Limit`, or 
`SubqueryAlias`).
   2. Added `find_explicit_sort` and `project_ordering` to traverse the 
physical execution plan down to `SortExec` through intermediate 
`ProjectionExec`, `GlobalLimitExec`, `LocalLimitExec`, and 
`CoalescePartitionsExec`.
   3. In `project_ordering`, use 
`datafusion_physical_plan::projection::update_expr` to translate physical sort 
expressions across intermediate projection schemas, truncating at the first 
missing leading key so only the valid ordered prefix is preserved.
   4. Added Rust unit tests in `physical_planner.rs` and sqllogictest cases in 
`copy.slt`.
   
   ## What is the testing strategy for this PR?
   
   - Unit tests in `datafusion/core/src/physical_planner.rs`:
     - `test_has_explicit_sort`
     - `test_project_ordering_and_find_explicit_sort`
   - Sqllogictests in `datafusion/sqllogictest/test_files/copy.slt`:
     - Verification of reproducer query (`COPY (SELECT id FROM (...) ORDER BY 
id, ts)`)
     - Verification of non-leading sort key projection (sort eliminated only 
when valid prefix is preserved)
     - Verification of column reordering and aliasing
   - Formatting and clippy checks (`cargo fmt -- --check`, `cargo clippy`).
   
   ## Are there any user-facing changes?
   
   No breaking API changes. Queries with explicit `ORDER BY` writing to `COPY 
... TO` will now properly preserve input ordering in `DataSinkExec` even when 
inner plans contain projections or limits.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to