kamilk-pal opened a new pull request, #27562:
URL: https://github.com/apache/datafusion/pull/27562

   ## Rationale for this change
   
   The memory-limited (spilling) fallback of `NestedLoopJoinExec`, added in 
54.0.0 by #21448, can panic or silently return incorrect results on 
`branch-54`. Fixing it on 54.x would take at least three bug fixes:
   
   - #24677: the build side is executed twice, which panics on a 
`RepartitionExec` and loses rows from file scans (#24676)
   - #24675: LEFT-type joins with a multi-partition probe side emit wrong 
unmatched rows (#22641)
   - #24746: unmatched probe-side rows are dropped after the last chunk (#24745)
   
   These sit on top of continued rework of the feature on `main`: #21636, 
#21833, #22038, #25004, #25533, #25542. More reports keep arriving, e.g. #25701 
(an empty result instead of spilling, reported on 55.1.0). All of it is first 
released in 56.0.0.
   
   Rather than backporting that series, this PR disables the fallback on 
`branch-54`. A nested loop join whose build side does not fit in memory fails 
with `ResourcesExhausted`, as it did before 54.0.0.
   
   ## What changes are included in this PR?
   
   - `ENABLE_MEMORY_LIMITED_EXECUTION = false` in `nested_loop_join.rs`, 
checked where `execute` decides whether it may fall back. The fallback code 
itself is unchanged.
   - Tests that need the fallback are `#[ignore]`d, a new test checks that 
every join type fails with `ResourcesExhausted`, and 
`nested_loop_join_spill.slt` expects the error.
   
   ## Are there any user-facing changes?
   
   Yes. On 54.x, a nested loop join whose build side exceeds the memory limit 
fails with `ResourcesExhausted` instead of spilling.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to