[
https://issues.apache.org/jira/browse/KAFKA-1546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14328298#comment-14328298
]
Jay Kreps commented on KAFKA-1546:
----------------------------------
Yeah that is true. I think we are in agreement that we want to express this in
terms of time not # messages.
The criteria I was proposing is "not caught up for N ms" where the definition
of not caught up is "reads to the end of the log".
I think what you are proposing is "will take more than N ms to catch up."
Originally I had thought a little about this. However this criteria is a lot
harder to calculate. In order to predict the time to catch up you need to
estimate the rate at which messages will be read in the future (e.g. if I am
1000 messages behind and reading at 500 msg/sec then it will take 2 seconds). I
was concerned that any estimate would be really fragile since the whole point
of a failure is that it changes this kind of rate in some way (because a
replica is slow, or messages got bigger, or whatever) so predictions off past
rates may be wrong once the (possibly) soft failure happens.
I think the motivation for the criteria I was proposing was that any caught up
reader should always be at the end of the log (that is the definition of caught
up) and if you go for a period of time without being at the end then likely you
won't get to the end soon. You could imagine some situation in which somehow
the follower was able to exactly keep up but was always one message behind the
end of the log in which case we would falsely failure detect the follower.
However I think this would be unlikely and failure detecting is actually
probably okay since you are exactly on the verge of overwhelmed (one byte per
second more throughput and you will be dead).
Let me know if you think that makes sense, I could definitely be convinced it
would be better a different way.
> Automate replica lag tuning
> ---------------------------
>
> Key: KAFKA-1546
> URL: https://issues.apache.org/jira/browse/KAFKA-1546
> Project: Kafka
> Issue Type: Improvement
> Components: replication
> Affects Versions: 0.8.0, 0.8.1, 0.8.1.1
> Reporter: Neha Narkhede
> Assignee: Aditya Auradkar
> Labels: newbie++
>
> Currently, there is no good way to tune the replica lag configs to
> automatically account for high and low volume topics on the same cluster.
> For the low-volume topic it will take a very long time to detect a lagging
> replica, and for the high-volume topic it will have false-positives.
> One approach to making this easier would be to have the configuration
> be something like replica.lag.max.ms and translate this into a number
> of messages dynamically based on the throughput of the partition.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
