[
https://issues.apache.org/jira/browse/KAFKA-1555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14072790#comment-14072790
]
Robert Withers commented on KAFKA-1555:
---------------------------------------
Jun, I see what you mean and perhaps there is no quantitative difference, I
really do not know without prototyping it. My instincts tell me that the
qualitative difference may be meaningful.
Let's consider the situation with 5 replicas and acks set to 3 versus 2
replicas and acks set to -1 as well as 3 shadows. Perhaps I am on the algebra
kick a bit much but the second arrangement means that we qualitatively have 2
well defined algebraic groups, the ISR for sure is consistent while the shadows
may be. Implementing the algebra as first class may have advantages. Take a
write chained ocap for the leader with 2 types of monitor ocaps: acking and
forwarding. Now the producer group constraint can be modeled: conservative or
opportunistic and the group constraint type can emit the right kind of ocap.
The forwarding ocap could be used to recast that mirror capability. Now we can
model the system algebraically, in first class objects: Metaprogramming of a
sort.
I think that we are talking about a 4 dimensional meta matrix, now: the front
consumer plane with translations and rotations for consumption and rebalance,
the 3rd dimension for ISR leader election and the 4th dimension for optimistic
shadowing and mirroring.
I have no evidence that any of this is beneficial or not. That's y'all's
decision to make.
Thank you for hearing me out.
> provide strong consistency with reasonable availability
> -------------------------------------------------------
>
> Key: KAFKA-1555
> URL: https://issues.apache.org/jira/browse/KAFKA-1555
> Project: Kafka
> Issue Type: Improvement
> Components: controller
> Affects Versions: 0.8.1.1
> Reporter: Jiang Wu
> Assignee: Neha Narkhede
>
> In a mission critical application, we expect a kafka cluster with 3 brokers
> can satisfy two requirements:
> 1. When 1 broker is down, no message loss or service blocking happens.
> 2. In worse cases such as two brokers are down, service can be blocked, but
> no message loss happens.
> We found that current kafka versoin (0.8.1.1) cannot achieve the requirements
> due to its three behaviors:
> 1. when choosing a new leader from 2 followers in ISR, the one with less
> messages may be chosen as the leader.
> 2. even when replica.lag.max.messages=0, a follower can stay in ISR when it
> has less messages than the leader.
> 3. ISR can contains only 1 broker, therefore acknowledged messages may be
> stored in only 1 broker.
> The following is an analytical proof.
> We consider a cluster with 3 brokers and a topic with 3 replicas, and assume
> that at the beginning, all 3 replicas, leader A, followers B and C, are in
> sync, i.e., they have the same messages and are all in ISR.
> According to the value of request.required.acks (acks for short), there are
> the following cases.
> 1. acks=0, 1, 3. Obviously these settings do not satisfy the requirement.
> 2. acks=2. Producer sends a message m. It's acknowledged by A and B. At this
> time, although C hasn't received m, C is still in ISR. If A is killed, C can
> be elected as the new leader, and consumers will miss m.
> 3. acks=-1. B and C restart and are removed from ISR. Producer sends a
> message m to A, and receives an acknowledgement. Disk failure happens in A
> before B and C replicate m. Message m is lost.
> In summary, any existing configuration cannot satisfy the requirements.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
