[
https://issues.apache.org/jira/browse/KAFKA-14145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jose Armando Garcia Sancio updated KAFKA-14145:
-----------------------------------------------
Description:
Typically, the HWM is increase after one round of Fetch requests from the
majority of the replicas. The HWM is propagated after another round of Fetch
requests. If the LEO doesn't change the propagation of the HWM can be delay by
one Fetch wait timeout (500ms).
Looking at the KafkaRaftClient implementation we would have to have an index
for both the fetch offset and the last sent high-watermark for that replica.
Another issue here is that we changed the KafkaRaftManager so that it doesn't
set the replica id when it is an observer/broker. Since the HWM is not part of
the Fetch request the leader would have to keep track of this in the
LeaderState.
{code:java}
val nodeId = if (config.processRoles.contains(ControllerRole)) {
OptionalInt.of(config.nodeId)
} else {
OptionalInt.empty()
} {code}
We would need to find a better solution for
https://issues.apache.org/jira/browse/KAFKA-13168 or improve the FETCH request
so that it includes the HWM.
was:
Typically, the HWM is increase after one round of Fetch requests from the
majority of the replicas. The HWM is propagated after another round of Fetch
requests. If the LEO doesn't change the propagation of the HWM can be delay by
one Fetch wait timeout (500ms).
Looking at the KafkaRaftClient implementation we would have to have an index
for both the fetch offset and the last sent high-watermark for that replica.
Another issue here is that we changed the KafkaRaftManager so that it doesn't
set the replica id when it is an observer/broker. Since the HWM is not part of
the Fetch request the leader would have to keep track of this in the
LeaderState.
val nodeId = if (config.processRoles.contains(ControllerRole)) \{
OptionalInt.of(config.nodeId)
} else \{
OptionalInt.empty()
}{{}}
We would need to find a better solution for
https://issues.apache.org/jira/browse/KAFKA-13168 or improve the FETCH request
so that it includes the HWM.
> Faster propagation of high-watermark in KRaft topic partitions
> --------------------------------------------------------------
>
> Key: KAFKA-14145
> URL: https://issues.apache.org/jira/browse/KAFKA-14145
> Project: Kafka
> Issue Type: Task
> Components: kraft
> Reporter: Jose Armando Garcia Sancio
> Assignee: Jose Armando Garcia Sancio
> Priority: Critical
> Fix For: 3.4.0
>
>
> Typically, the HWM is increase after one round of Fetch requests from the
> majority of the replicas. The HWM is propagated after another round of Fetch
> requests. If the LEO doesn't change the propagation of the HWM can be delay
> by one Fetch wait timeout (500ms).
> Looking at the KafkaRaftClient implementation we would have to have an index
> for both the fetch offset and the last sent high-watermark for that replica.
> Another issue here is that we changed the KafkaRaftManager so that it doesn't
> set the replica id when it is an observer/broker. Since the HWM is not part
> of the Fetch request the leader would have to keep track of this in the
> LeaderState.
> {code:java}
> val nodeId = if (config.processRoles.contains(ControllerRole)) {
> OptionalInt.of(config.nodeId)
> } else {
> OptionalInt.empty()
> } {code}
> We would need to find a better solution for
> https://issues.apache.org/jira/browse/KAFKA-13168 or improve the FETCH
> request so that it includes the HWM.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
