HappenLee opened a new pull request, #67894:
URL: https://github.com/apache/doris/pull/67894
### What problem does this PR solve?
Issue Number: N/A
Related PR: N/A
Problem Summary:
Filtering `rank()` or `dense_rank()` through PartitionTopN can silently lose
rows when the last qualifying peer group spans output batches. With
`batch_size=4` and five rows sharing the lowest order key, `rank <= 1` returns
only four rows. The rank reader's deferred EOS check treats a satisfied rank
counter as the end of the peer group, causing final output and intermediate
sink pruning to stop too early.
Report EOS only when the next peer group is outside the rank limit or the
merge queue is exhausted. Preserve the comparison cursor and counters across
full output batches.
Add parameterized BE tests for batch boundaries, multiple merge cursors,
exhausted input, and rank versus dense_rank limits. SQL regression coverage
includes global and partitioned windows, a two-row boundary group starting
mid-batch, ROW_NUMBER, and intermediate pruning with 21,005 qualifying peers.
The large case uses 1,000-row batches so it crosses the 20,000-row pruning
threshold before the final input batch; small boundary cases use four-row
batches.
### Release note
Fix missing rows in rank() and dense_rank() filters when PartitionTopN peer
groups span output batches.
### Check List (For Author)
- Test:
- Unit Test: 11 ASAN tests passed in `PartitionSorterTest` and
`RankAlgorithms/PartitionSorterRankTest` through `run-be-ut.sh`.
- Regression test: `test_partition_topn_rank_batch` and existing
`test_partition_topn` both passed with zero failures, fatal errors, or skips
through `run-regression-test.sh`.
- Expected output was generated by the regression runner with
PartitionTopN disabled, verified in a normal oracle run, and then matched by
the fixed optimized path. The old optimized binary fails the new suite at
`rank_global`, omitting a peer row.
- BE ASAN build, clang-format 16.0.6, build hygiene, and clang-tidy
passed. Local BE validation used `USE_OPENMP=FALSE` in the build cache after an
OpenBLAS/OpenMP initialization crash, and a 64 GB test-instance memory limit
for ASAN. These environment settings are excluded from the patch.
- Behavior changed: Yes. Preserve all rows in qualifying rank/dense_rank
peer groups across batches.
- Does this need documentation: No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
