HappenLee opened a new pull request, #69184:
URL: https://github.com/apache/doris/pull/69184
### What problem does this PR solve?
Issue Number: N/A
Related PR: #68961
A full UNPARTITIONED broadcast block currently gets copied into the
serializer's MutableBlock before it is serialized, even when every destination
is remote and there are no pending rows. This adds a full-column copy and an
accumulation buffer to an already complete batch.
Serialize the input directly when all channels are remote, the accumulation
buffer is absent or empty, and the input reaches the batch row limit.
Serialization finishes synchronously; asynchronous sends retain only the
existing owned BroadcastPBlockHolder. Both paths share protobuf encoding, queue
admission, the broadcast loop, error handling, and statistics. Pending rows and
small inputs keep their existing batching behavior; local and mixed exchanges
keep their current path. Empty EOS and low-memory buffer release are preserved.
### Release note
Reduce intermediate copying and accumulation-buffer memory when broadcasting
full exchange blocks to remote destinations.
### Check List (For Author)
- Test:
- Unit tests cover full/oversized batches, small and pending inputs,
input reuse, String/Nullable/Const/Array columns with no
compression/LZ4/Snappy, single/multiple remote receivers, mixed destinations,
empty EOS, low-memory cleanup, receiver EOF, send failure, and broadcast
limiter accounting/backpressure.
- ASAN: `ENABLE_PCH=OFF ENABLE_UNITY_BUILD=ON ./run-be-ut.sh -j 48 --run
--filter='*ExchangeSinkOperatorTest*:*ExchangeSinkTest*:*BroadcastProtobufTest*'`
— all 32 tests passed, including 8 new tests.
- clang-format 16.0.6, build hygiene, and `git diff --check`: passed.
- clang-tidy: all checks passed for the changed lines; the test file
reported no warnings.
- No distributed SQL or end-to-end performance benchmark was run. Tests
verify that eligible batches allocate no accumulation buffer and do not enter
MergeBlockTime; no measured throughput improvement is claimed.
- Behavior changed: No. SQL results and the exchange protocol are unchanged;
existing constant-column serialization may use a more compact representation
than the materialized merge path.
- Does this need documentation: No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
