lzydmxy opened a new pull request, #65584:
URL: https://github.com/apache/doris/pull/65584
INSERT OVERWRITE ... PARTITION(*), and any load that opens incremental
streams at runtime (auto-partition create / auto-detect overwrite), could hang
forever. On the receiver, LoadStream handled CLOSE_LOAD by registering the
stream for deferred close and checking all-received in two separate lock
scopes, with a _report_result network IO in between. Under concurrent
CLOSE_LOADs the last stream (all_closed=true) could drain and clear
_closing_stream_ids before a counted-but-late stream's push_back, orphaning
that stream so it was never StreamClose'd. Its on_closed never fired, the sink
close_wait waited forever, the fragment never reported done, and the FE latch
never released.
A second, related hazard: the all_closed thread StreamClose'd another stream
while that stream's EOS (_report_result) had not been sent yet, so the sender
saw on_closed with _is_eos=0 and reported "Stream closed without EOS".
Both share one root: "the close action" and "this stream's EOS was sent" are
not ordered, and "register" and "all-received check" are not under the same
lock. This introduces mark_eos_sent_and_collect(): register + the all-received
check + collect-to-close run under one lock, a stream is registered only after
its own EOS was sent, and a write-once _all_close_load_received latch makes any
thread arriving after all-received drain the remaining streams, so no orphan
survives any interleaving. Fencing (closing incremental streams together, after
all non-incremental ones) is preserved.
Adds a docker regression suite plus an inert debug point
(LoadStream.close_load.delay_incremental_register) that deterministically opens
the race window: the buggy binary hangs, the fixed binary completes.
### What problem does this PR solve?
Issue Number: close #65582
Related PR: #56120
Problem Summary:
`INSERT OVERWRITE ... PARTITION(*)`, and any load that opens incremental
streams at runtime (auto-partition create / auto-detect overwrite), could hang
forever until the 4h `insert_load_default_timeout_second`. The root cause is a
receiver-side load-stream close race in `_dispatch`'s `CLOSE_LOAD` handling:
`close()` (count + decide `all_closed`) and the deferred-close bookkeeping ran
in two separate lock scopes with a `_report_result` network IO in between, so a
counted-but-late stream could be registered after the last stream already
drained and cleared `_closing_stream_ids`, leaving it never `StreamClose`d.
That orphaned stream's `on_closed` never fired, the sink `close_wait` waited
forever, the fragment never reported done, and the FE latch never released. A
second hazard in the same structure closed a stream before its EOS was sent,
surfacing as `Stream closed without EOS`.
`_closing_stream_ids` was introduced in #56120; this PR fixes the race it
left behind. Latest `master` still has the buggy structure.
### Release note
Fix a permanent hang of `INSERT OVERWRITE ... PARTITION(*)` and other loads
that open incremental streams at runtime, caused by a receiver-side load-stream
close race.
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [x] Regression test
- [ ] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [x] Yes. Fixes a hang; loads that previously blocked forever now
complete. No change on the normal success path.
- Does this need documentation?
- [x] No.
- [ ] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR should
merge into -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
