[
https://issues.apache.org/jira/browse/KAFKA-1546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14328318#comment-14328318
]
Neha Narkhede commented on KAFKA-1546:
--------------------------------------
bq. For example a replica could be catching up quickly but the
"replica.lag.max.ms" counter would still increase until it fully catches up and
then it will abruptly drop to zero.
What we want to measure is not *exactly* how slow it is but express lag in
terms of maximum time spent not catching up with the leader. The check that Jay
mentions can be improved a little. Basically, if the replica is at the log end
offset, then we don't want to check for lagBegin at all. The first time a
replica starts lagging, you set the lagBegin to current time. From there on,
you only reset it to -1 if it reaches log end offset.
This will remove a replica that keeps fetching but is unable to catch up with
the leader for "replica.lag.max.ms".
So the check is more like-
{code}
if(!fetchedData.readToEndOfLog) {
if(lagBegin == -1) {
this.lagBegin = System.currentTimeMillis()
}
} else {
this.lagBegin = -1
}
{code}
Then the liveness criteria is
partitionLagging = this.lagBegin > 0 && System.currentTimeMillis() -
this.lagBegin > REPLICA_LAG_TIME_MS
In order to do this, LogReadResult might have to return the log end offset as
well.
> Automate replica lag tuning
> ---------------------------
>
> Key: KAFKA-1546
> URL: https://issues.apache.org/jira/browse/KAFKA-1546
> Project: Kafka
> Issue Type: Improvement
> Components: replication
> Affects Versions: 0.8.0, 0.8.1, 0.8.1.1
> Reporter: Neha Narkhede
> Assignee: Aditya Auradkar
> Labels: newbie++
>
> Currently, there is no good way to tune the replica lag configs to
> automatically account for high and low volume topics on the same cluster.
> For the low-volume topic it will take a very long time to detect a lagging
> replica, and for the high-volume topic it will have false-positives.
> One approach to making this easier would be to have the configuration
> be something like replica.lag.max.ms and translate this into a number
> of messages dynamically based on the throughput of the partition.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
