unbridled-41 opened a new pull request, #11046:
URL: https://github.com/apache/rocketmq/pull/11046
Closes #11045
### Problem / Evidence
`DefaultBrokerHeartbeatManager#onBrokerHeartbeat` never rebinds the channel
of an existing `BrokerLiveInfo` — the `prev != null` branch only refreshes the
timestamp, timeout, election priority and epoch
(`DefaultBrokerHeartbeatManager.java:133-142`), so the live entry stays pinned
to the broker's **first-ever** channel.
When the broker-to-controller connection dies half-open (or the
controller-side close event for the old channel arrives late), the broker
client reconnects first and keeps heartbeating on a new channel. When the old
channel eventually fires
`channelInactive`/`onChannelException`/`onChannelIdle`,
`BrokerHousekeepingService` (wired as the controller `ChannelEventListener` in
`ControllerManager`, passed to `DLedgerServer`) calls
`onBrokerChannelClose(oldChannel)`, which matches the stale stored channel,
**removes the live entry of a broker that is healthy on the new channel** and
fires `notifyBrokerInActive`. `ControllerManager#onBrokerInactive` then checks
`getReplicaInfo` and, because the evicted broker is the current master, calls
`triggerElectMaster` — a spurious failover (epoch bump, possible master change,
sync-state reset) for a healthy broker-set.
Deterministic regression test
`DefaultBrokerHeartbeatManagerTest#testStaleChannelCloseDoesNotEvictReRegisteredBroker`
fails on unmodified `develop`:
```
DefaultBrokerHeartbeatManagerTest.testStaleChannelCloseDoesNotEvictReRegisteredBroker:74
A stale channel close must not evict a broker alive on a new channel
Tests run: 2, Failures: 1, Errors: 0, Skipped: 1 (develop @ ff8f6f74c,
2026-09-05)
```
### Root cause / Fix
The heartbeat path keeps the stale channel reference forever. This is
inconsistent with the NameServer implementation, where the broker's channel is
refreshed on every registration heartbeat.
Fix: rebind the live entry to the current channel in `onBrokerHeartbeat`
when a non-null channel arrives, so `onBrokerChannelClose` only removes entries
whose stored channel is the one that actually closed:
```java
// Rebind the live entry to the current channel, otherwise a close event of
a stale
// channel would evict a broker that is already alive on a new channel
if (channel != null && channel != prev.getChannel()) {
prev.setChannel(channel);
}
```
### Priority
PRIORITY = 76（影响 30 + 波及范围 14 + 可复现 18 + 维护价值 14），FIX_CONFIDENCE = 88。
- 影响 30/40: spurious master election / epoch bump fencing a healthy master —
an availability and correctness event for every controller-mode deployment (the
default dLedger controller uses this heartbeat manager).
- 波及范围 14/20: all controller-mode deployments; the mechanism (heartbeat +
channel lifecycle) runs continuously.
- 可复现 18/20: deterministic two-channel unit test; the reconnect-before-close
ordering is a routine network scenario.
- 维护价值 14/20: same bug family as the nameserver stale-channel issues; small,
well-bounded fix aligning controller with nameserver behavior.
### Tests
- Regression test added:
`DefaultBrokerHeartbeatManagerTest#testStaleChannelCloseDoesNotEvictReRegisteredBroker`
(fails before the fix as shown above).
- After the fix (`develop @ ff8f6f74c` + this change, 2026-09-05):
```
mvn -pl controller test -Dtest=DefaultBrokerHeartbeatManagerTest
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0 — BUILD SUCCESS
```
- Full controller module: `mvn -pl controller test` — all tests pass except
the pre-existing baseline failure `ReplicasInfoManagerTest#testSerialize`
(JDK-21 `InaccessibleObjectException: Unable to make field ... AtomicLong.value
accessible`), which reproduces identically on unmodified `develop @ ff8f6f74c`
and is unrelated to this change.
### Risk
Very low. The added branch only updates a mutable field that was previously
never refreshed after creation; heartbeats without a channel (the existing test
passes `null`) keep the old behavior. The close-detection semantics
(`getChannel() == channel` identity match) are unchanged — they simply now
compare against the current channel. The jRaft heartbeat manager keeps its own
raft-routed close handling and is untouched.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
