[
https://issues.apache.org/jira/browse/KAFKA-1555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14087795#comment-14087795
]
Gwen Shapira commented on KAFKA-1555:
-------------------------------------
We are talking about topic level, and I completely agree with your
interpretation.
I don't think its odd to separate some of the reliability concerns between
admin (who creates topics and configures them) and the developer (who controls
specific requests). The admin already controls some of the safety guarantees by
controlling the number of replicas.
At the risk of using a not-100% relevant analogy, I'd like to compare the
behavior to Oracle's commits (my favorite transaction log):
A developer decides when to "commit" a transaction. "Commits" are always
synchronous and always checkpoint to disk and therefore will take time, and
they guarantee durability - a committed transaction will not disappear without
some kind of a disaster.
The admin controls how much of a disaster can make a commit disappear - the log
can be written to a single local disk (high chance of disaster), an HA SAN
(longer writes but safer), SAN + Disk (waiting for both to ack), Disk + SSD
(waiting for one to ack), disk + remote replica (not waiting for replica), disk
+ replica (wait for replica), etc, etc. All this is managed by the DBA. The
developer just says "commit", meaning "I want durability and I'm willing to
wait", trusting the admins definition of durability.
Anyway, all this to show that controlling durability in two different places is
pretty normal in many other communities. We won't be "odd" by supporting this.
> provide strong consistency with reasonable availability
> -------------------------------------------------------
>
> Key: KAFKA-1555
> URL: https://issues.apache.org/jira/browse/KAFKA-1555
> Project: Kafka
> Issue Type: Improvement
> Components: controller
> Affects Versions: 0.8.1.1
> Reporter: Jiang Wu
> Assignee: Neha Narkhede
>
> In a mission critical application, we expect a kafka cluster with 3 brokers
> can satisfy two requirements:
> 1. When 1 broker is down, no message loss or service blocking happens.
> 2. In worse cases such as two brokers are down, service can be blocked, but
> no message loss happens.
> We found that current kafka versoin (0.8.1.1) cannot achieve the requirements
> due to its three behaviors:
> 1. when choosing a new leader from 2 followers in ISR, the one with less
> messages may be chosen as the leader.
> 2. even when replica.lag.max.messages=0, a follower can stay in ISR when it
> has less messages than the leader.
> 3. ISR can contains only 1 broker, therefore acknowledged messages may be
> stored in only 1 broker.
> The following is an analytical proof.
> We consider a cluster with 3 brokers and a topic with 3 replicas, and assume
> that at the beginning, all 3 replicas, leader A, followers B and C, are in
> sync, i.e., they have the same messages and are all in ISR.
> According to the value of request.required.acks (acks for short), there are
> the following cases.
> 1. acks=0, 1, 3. Obviously these settings do not satisfy the requirement.
> 2. acks=2. Producer sends a message m. It's acknowledged by A and B. At this
> time, although C hasn't received m, C is still in ISR. If A is killed, C can
> be elected as the new leader, and consumers will miss m.
> 3. acks=-1. B and C restart and are removed from ISR. Producer sends a
> message m to A, and receives an acknowledgement. Disk failure happens in A
> before B and C replicate m. Message m is lost.
> In summary, any existing configuration cannot satisfy the requirements.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
