Janus Chow created RATIS-2685:
---------------------------------
Summary: Follower gauges are stale after LogAppender restart
Key: RATIS-2685
URL: https://issues.apache.org/jira/browse/RATIS-2685
Project: Ratis
Issue Type: Bug
Reporter: Janus Chow
Assignee: Janus Chow
{{LeaderStateImpl.restart(LogAppender)}} creates a new {{FollowerInfo}} for the
same follower and calls {{LogAppenderMetrics.addFollowerGauges}} again. The old
follower gauges are never removed:
* {{follower_<id>_next_index}}
* {{follower_<id>_match_index}}
* {{follower_<id>_rpc_response_time}}
{{RatisMetricRegistry.gauge(..)}} delegates to Dropwizard
{{{}MetricRegistry.gauge(..){}}}, which returns the existing gauge instead of
replacing it. So after a restart these gauges keep reading the stopped
{{FollowerInfo}} and never change again.
In Apache Ozone, this made a pipeline look like it was lagging more and more:
{{appliedIndex - follower_next_index}} kept growing, even though all followers
were in sync.
Proposed fix: remove the follower gauges when senders are stopped and before
re-registering them.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
