goutamadwant opened a new pull request, #24482: URL: https://github.com/apache/datafusion/pull/24482
## Which issue does this PR close? - Closes #24469. ## Rationale for this change `UnnestExec` currently records its baseline output metrics before `BatchSplitStream` divides an oversized result. For example, unnesting 25 rows with a batch size of 10 emits batches containing `[10, 10, 5]` rows, but reports `output_batches=1` instead of `3`. This makes the metric disagree with the batches observed by downstream operators and by users inspecting execution metrics. ## What changes are included in this PR? - Move baseline output observation to the final stream boundary by wrapping `BatchSplitStream` with the existing `ObservedStream`. - Keep `elapsed_compute` measurement inside `UnnestStream`, where the unnest work is performed. - Add a regression test that verifies the emitted batch sizes and the corresponding `output_batches` and `output_rows` metrics. As a result, baseline output metrics, including `output_bytes`, are recorded from the batches emitted by the final stream. As documented for `BaselineMetrics`, `output_bytes` may overestimate memory when sliced batches share underlying buffers. ## Are these changes tested? Yes. - The regression test fails before the production change with `output_batches=1` and passes afterward with `output_batches=3`. - All 13 `unnest` tests pass. - All 1,760 `datafusion-physical-plan` unit tests and its documentation tests pass. - The extended workspace test command from `AGENTS.md` passes, including all 503 SQL logic test files. - All 107 `datafusion-cli` unit and integration tests pass. - `cargo fmt --all -- --check` passes. - `cargo clippy --all-targets --all-features -- -D warnings` passes. - Rust documentation builds with warnings denied. ## Are there any user-facing changes? Yes. `UnnestExec` now reports baseline output metrics at the consumer-visible output boundary, so `output_batches` matches the number of emitted batches. Query results, row ordering, batch boundaries, plans, and public APIs are unchanged. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
