Jason,
As Neha said, what you said is possible, but may require a more careful
design. For example, what if the followers don't catch up with the leader
quickly? Do we want to wait forever or up to some configurable amount of
time? If we do the latter, we may still lose data during controlled
shut
Thanks Neha for continued insight
What you describe as a possible solution is what I was thinking (although I
wasn't as concerned as maybe I should be with the added delay of the new
leader delaying processing new requests while it finishes consuming from
the old leader, and communicates back
Ok, so if I initiate a controlled shutdown, in which all partitions that a
shutting down broker is leader of get transferred to another broker, why
can't part of that controlled transfer of leadership include ISR
synchronization, such that no data is lost? Is there a fundamental reason
why that is
On Sun, Oct 6, 2013 at 4:08 AM, Neha Narkhede wrote:
> Does the
> leader just wait for the followers in the ISR to consume?
>
> That's right. Until that is done, the producer does not get an ack back. It
> has an option of retrying if the previous request times out or fails.
>
>
Ok, so if I initia
Does the
leader just wait for the followers in the ISR to consume?
That's right. Until that is done, the producer does not get an ack back. It
has an option of retrying if the previous request times out or fails.
A separate question, can the request.required.acks be set to a higher
positive integ
Thanks for the explanation Neha.still holding out hope.
So, if request.required.acks=-1, how does the leader confirm that the other
brokers have consumed the message, before acking to the producer? Does the
leader just wait for the followers in the ISR to consume? Or does the
leader have
Shouldn't this be part of the contract? It should be able to make sure
this happens before shutting down, no?
The leader writes messages to its local log and then the replicas consume
messages from the leader and write those to their local logs. If you set
request.required.acks=1, the ack is sent
Neha,
I'm not sure I understand. I would have thought that if the leader
acknowledges receipt of a message, and is then shut down cleanly (with
controlled shutdown enabled), that it would be able to reliably persist any
in memory buffered messages (and replicate them), before shutting down.
Shou
The occasional single message loss could happen since
required.request.acks=1 and the leader is shut down before the follower
gets a chance to copy the message. Can you try your test with num acks set
to -1 ?
Thanks,
Neha
On Oct 4, 2013 1:21 PM, "Jason Rosenberg" wrote:
> All,
>
> I'm having an
