bobhan1 opened a new pull request, #69450:
URL: https://github.com/apache/doris/pull/69450
### What problem does this PR solve?
Issue Number: N/A
Related PR: N/A
Problem Summary:
In FQDN mode, a BE Pod restart can leave FE with a stale hostname-to-IP
mapping. If another BE takes the old IP, a new heartbeat connection can reach
that process. The process can return an identity error through a successful
Thrift RPC, and FE returns the socket to the pool. Later heartbeats reuse it
even after DNS has converged. A wrong process can also accept the heartbeat
while its own DNS view is stale.
Resolve heartbeat destinations through the existing FE DNSCache and borrow
connections using the resolved IP, while preserving the original backend
hostname and ID in TMasterInfo. Close the previous endpoint's idle pool when
the address changes, and clean up pools for removed backends. Invalidate
connections on unsuccessful heartbeat responses so the existing retry can
reconnect. Recovery follows the existing DNSCache refresh policy.
### Release note
Fix persistent backend heartbeat failures after BE Pod IP changes in FQDN
mode.
### Check List (For Author)
- Test
- [ ] Regression test
- [x] Unit Test
- [ ] Manual test
- [ ] No need to test or manual test.
- Behavior changed:
- [ ] No.
- [x] Yes. Heartbeats switch connections when the cached IP changes and
discard connections after application errors.
- Does this need documentation?
- [x] No.
- [ ] Yes.
Validation:
All 39 targeted FE tests pass with 0 failures, errors, or skips. Both
changed Java files pass Checkstyle 9.3 using the repository's Checkstyle
configuration. The FE UT script finishes with BUILD SUCCESS.
```bash
./run-fe-ut.sh --run
org.apache.doris.system.BackendHeartbeatHandlerTest,org.apache.doris.system.HeartbeatMgrTest,org.apache.doris.common.GenericPoolTest,org.apache.doris.common.DNSCacheTest,org.apache.doris.rpc.BackendServiceProxyTest
```
The new tests cover stable-address reuse, preserved FQDN identity, rejected
replies, RPC exceptions, DNS lookup failure, IP mode, removed-backend cleanup,
and address changes. Two tests use live loopback Thrift servers and wait for
the old connection to close, including a healthy old connection that continues
accepting heartbeats.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
