[ 
https://issues.apache.org/jira/browse/KAFKA-1555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14147474#comment-14147474
 ] 

Sriram Subramanian commented on KAFKA-1555:
-------------------------------------------

1. Great. Not supporting all values above ack > 1 is a good step. We are 
essentially not using it as an integer any more. I would still  love it to be 
made more explicit with an enum for clarity.

2. Also, by setting ack = -1 and min_isr = 2, we still do not avoid data loss 
when one broker goes down. The issue is the way we select a leader. When a 
request was written to the leader, the min_isr check could have succeeded and 
we would have written to min_isr - 1 number of replicas. However, the replicas 
could subsequently go out of the isr. When the leader fails after that, we 
would have an unclean leader election and select any replica as the leader and 
it could be one that was lagging. 

To completely guarantee no data loss, we would need to do the following
a. Ensure logs do not diverge on unclean leader elections
b. Choose the broker with the longest log as the leader

3. We may have not documented ack > 1 but since it is an integer, there are 
chances somebody could be using it. In such a case this could be a backwards 
incompatible change. It would be worth mentioning it in the release notes.

4. Long term, I think the min_isr config should be in the API. This gives 
better control per message and explicitly lets the caller know what guarantees 
they get.

> provide strong consistency with reasonable availability
> -------------------------------------------------------
>
>                 Key: KAFKA-1555
>                 URL: https://issues.apache.org/jira/browse/KAFKA-1555
>             Project: Kafka
>          Issue Type: Improvement
>          Components: controller
>    Affects Versions: 0.8.1.1
>            Reporter: Jiang Wu
>            Assignee: Gwen Shapira
>             Fix For: 0.8.2
>
>         Attachments: KAFKA-1555.0.patch, KAFKA-1555.1.patch, 
> KAFKA-1555.2.patch, KAFKA-1555.3.patch, KAFKA-1555.4.patch
>
>
> In a mission critical application, we expect a kafka cluster with 3 brokers 
> can satisfy two requirements:
> 1. When 1 broker is down, no message loss or service blocking happens.
> 2. In worse cases such as two brokers are down, service can be blocked, but 
> no message loss happens.
> We found that current kafka versoin (0.8.1.1) cannot achieve the requirements 
> due to its three behaviors:
> 1. when choosing a new leader from 2 followers in ISR, the one with less 
> messages may be chosen as the leader.
> 2. even when replica.lag.max.messages=0, a follower can stay in ISR when it 
> has less messages than the leader.
> 3. ISR can contains only 1 broker, therefore acknowledged messages may be 
> stored in only 1 broker.
> The following is an analytical proof. 
> We consider a cluster with 3 brokers and a topic with 3 replicas, and assume 
> that at the beginning, all 3 replicas, leader A, followers B and C, are in 
> sync, i.e., they have the same messages and are all in ISR.
> According to the value of request.required.acks (acks for short), there are 
> the following cases.
> 1. acks=0, 1, 3. Obviously these settings do not satisfy the requirement.
> 2. acks=2. Producer sends a message m. It's acknowledged by A and B. At this 
> time, although C hasn't received m, C is still in ISR. If A is killed, C can 
> be elected as the new leader, and consumers will miss m.
> 3. acks=-1. B and C restart and are removed from ISR. Producer sends a 
> message m to A, and receives an acknowledgement. Disk failure happens in A 
> before B and C replicate m. Message m is lost.
> In summary, any existing configuration cannot satisfy the requirements.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Reply via email to