matanper opened a new pull request, #70762:
URL: https://github.com/apache/doris/pull/70762
### What problem does this PR solve?
Issue Number: close #70382
Related PR: #59904, #69423, #69450
Problem Summary:
Since #59904 the FE dials the IP that `DNSCache` resolved for a backend.
`DNSCache.get()` is `computeIfAbsent`, and only the periodic task re-resolves
(every 60 s, first run after 120 s). `BackendServiceProxy.getProxy` rebuilds a
non-normal channel, but to the same stale IP. On top of that,
`NetUtils.getIpByHost` goes through `InetAddress`, whose own cache keeps
answers for `networkaddress.cache.ttl=60` (set in `DorisFE`), so even a forced
lookup can return the old IP for another minute. The heartbeat client resolves
the hostname through the same JVM cache.
When a BE restarts under the same hostname with a new IP (e.g. a Kubernetes
pod), every query that schedules a fragment on it fails with `send fragments
failed ... UNAVAILABLE: io exception` for up to a minute or more after the BE
is alive again. In a rolling restart this happens on every BE rejoin. #69450
fixed this for heartbeat connections only.
This is the narrow, default-on FE DNS part of #69423 (which also carries
graceful-restart, Stream Load and BE changes), with the first-lookup race from
its latest review handled, plus eviction of the JVM DNS entry, which #69423
does not do.
#### Fix
- **`DNSCache.invalidate(host)`** marks the host for a forced lookup on its
next `get()`:
- lookups for a host are serialized with `compute`, so an older result
cannot overwrite a newer one;
- a failed lookup (exception or empty result) keeps the last known IP;
- an invalidation that arrives while the host's first lookup is still
running is kept, so the next `get()` resolves again (the race found in #69423's
review);
- warm readers wait for an in-flight forced lookup instead of returning
the address it replaces;
- the periodic refresh resolves **outside** the map lock (a slow resolver
does not block other hosts in the same bin) and publishes only if no other
lookup published for the host since it started, tracked by a per-host version
bumped under the map lock. An address comparison would not be enough: a newer
lookup may confirm the same address the refresh started from. A refresh does
not consume a pending invalidation.
- A forced lookup first evicts the host from the JVM `InetAddress` cache
(`NetUtils.evictJvmDnsCache`, by reflection; `start_fe.sh` already passes
`--add-opens=java.base/java.net=ALL-UNNAMED`, and the fe-core surefire
`argLine` now does too). If the field is not reachable, nothing changes: the
entry expires by its TTL as before.
- **`DNSCache.invalidateAfterRpcFailure(host, failedIp)`** does nothing when
the cache already moved past `failedIp` (a late failure from an old channel),
and allows at most one forced lookup per host per second, so a BE that keeps
refusing connections at an unchanged address does not turn every failed RPC
into a DNS query.
- **`BackendServiceProxy.invalidateForAll(address)`** drops the cached
channel to the address from every proxy instance in `Holder` and shuts it down
without waiting, so the heartbeat or replay thread never blocks.
#### Triggers
1. **RPC failure.** A gRPC client interceptor on every
`BackendServiceClient` channel calls `invalidateAfterRpcFailure` when a call
closes with `UNAVAILABLE`, or with `DEADLINE_EXCEEDED` while the channel is not
`READY` (how calls end when the old address silently drops packets). A deadline
on a `READY` channel is a slow call to a reachable BE and is ignored;
application/query errors are never a trigger. Because it sits on the channel,
it covers fragment sends and every other backend RPC, including failures that
surface asynchronously through `Future.get()` in `Coordinator` /
`PipelineExecutionTask`. Only the DNS entry is invalidated; `getProxy` rebuilds
the channel only if the new lookup returns a different IP.
2. **Failing channel.** When `getProxy` finds the cached channel in a
non-normal state, it claims a forced lookup (same throttle) and always reads
the cache again before rebuilding, so it uses a lookup already marked by a
racing RPC failure even when the throttle drops its own claim. An empty DNS
result in FQDN mode is re-resolved on the next RPC instead of staying cached
until the periodic refresh.
3. **Heartbeat.** `Backend.handleHbResponse` calls
`invalidateLocalConnections()` (DNS invalidation plus `invalidateForAll` on the
brpc address) when an OK heartbeat shows the BE back alive or with a new start
time. It runs on the master and on every follower's heartbeat replay (each FE
has its own DNS cache and channels), and is skipped on the checkpoint thread so
historical replay never touches serving connections. A failed heartbeat on the
master evicts the host's JVM DNS entry, so the next heartbeat reaches a BE that
moved.
No new config. When the IP did not change, the cost is one DNS lookup, plus
one new channel on a BE restart.
Out of scope (as suggested in the issue triage): transparent RPC retries,
BE-to-BE exchange caches (BE side), Stream Load, graceful-shutdown admission.
#### Before / after (restart at a new IP)
This is an availability fix, not a throughput change, so the measurement is
time to recovery. Setup: a local FE + BE in FQDN mode
(`enable_fqdn_mode=true`), BE added as `fedns-be.test` via `/etc/hosts`.
Restart simulation: stop the BE, make the old IP (127.0.0.2) reset connections
to the BE ports with iptables, remap the hostname to 127.0.0.3, start the BE.
Query loop at 4/s (`select count(*) ... where v + rand() > 0`, never served
from cache). T0 = the BE accepts connections at the new IP. "Worst case"
restarts the BE 4 s after a periodic DNS refresh. These runs were done with the
same FE change on a 4.1.4-based build (the touched FE code is identical between
4.1.4 and master apart from the conflicts noted below).
| FE | restart timing | errors after T0 | `UNAVAILABLE` after T0 | first OK
query after T0 |
|---|---|---|---|---|
| before | worst case | 215 | 105 | 57.7 s |
| before | arbitrary | 37 | 37 | 10.4 s |
| after | worst case | 36 | 0 | 9.7 s |
| after | arbitrary | 0 | 0 | 0.2 s |
Blackhole variant (old IP drops packets): before 70.3 s to the first OK
query (3 queries hung for the loop's 20 s timeout), after 8.8 s.
The remaining errors after the fix are all `tablet has no queryable
replicas` while the scheduler still has the BE marked down; that window is the
same when a BE restarts at an unchanged IP and is not changed here.
Port notes vs. master: master dropped `channelConfigVersion` in
`BackendServiceClient` and moved the fe-core surefire flags to
`${fe.ut.jvm.args}`; the change is adapted to both.
### Release note
Fix FE sending query RPCs to the old IP of a backend that restarted on a new
IP under the same hostname (FQDN mode) for up to a minute or more.
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [x] Unit Test
- [x] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
New: `DNSCacheInvalidationTest` (10) and `BackendRestartReconnectTest`
(11): invalidate then re-resolve; a failed forced lookup keeps the last IP; a
stale periodic refresh does not overwrite a newer lookup, including one that
confirmed the same address; the refresh does not hold the lock during DNS;
invalidation during the first lookup; warm readers wait for a forced lookup;
RPC-failure invalidation scoped to the failing IP and throttled; JVM
`InetAddress` cache eviction of `localhost`; `getProxy` rebuilds to the new IP
after invalidation and for a `TRANSIENT_FAILURE` channel; a real `UNAVAILABLE`
RPC (closed port) forces a lookup; `DEADLINE_EXCEEDED` while connecting (socket
that never completes the HTTP/2 handshake) forces a lookup and on a `READY`
channel (in-process server that never answers) does not; a racing RPC failure's
lookup is used when the throttle drops `getProxy`'s claim; heartbeat with a new
start time, follower replay and recovery after failed heartbeats drop ch
annels and re-resolve; checkpoint replay leaves them alone. Each test was
checked to fail with the corresponding piece of the fix reverted.
`./run-fe-ut.sh --run` on master + this PR: `DNSCacheInvalidationTest`,
`BackendRestartReconnectTest`, `DNSCacheTest`, `BackendServiceProxyTest`,
`BackendServiceClientTest`, `HeartbeatMgrTest`, `BackendHeartbeatHandlerTest`,
`BackendRemoteSpillBytesTest`: 61 tests, 0 failures, 0 skipped.
- Behavior changed:
- [ ] No.
- [x] Yes. The FE re-resolves a backend hostname on demand after an
unreachable RPC or a BE restart/recovery heartbeat, instead of only every 60 s,
and bypasses the JVM DNS cache for those lookups.
- Does this need documentation?
- [x] No.
- [ ] Yes.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]