Re: Kafka send() call, ack, and message writes to the filesystem

2020-01-16 Thread Anindya Haldar
Does an acknowledgement in this scenario guarantee persistence of the message in the logs of the ISRs, including the leader? Sincerely, Anindya > On Jan 16, 2020, at 3:49 PM, M. Manna wrote: > > Andiya, > > On Wed, 15 Jan 2020 at 21:59, Anindya Haldar > wrote: > >> Okay, let’s say >> >> -

Re: Kafka send() call, ack, and message writes to the filesystem

2020-01-16 Thread M. Manna
Andiya, On Wed, 15 Jan 2020 at 21:59, Anindya Haldar wrote: > Okay, let’s say > > - the application is using a non-transactional producer, shared across > multiple threads > - the linger.ms and buffer.memory is non-zero, and so is batch.size such > that messages are actually batched > - the repl

Re: Kafka send() call, ack, and message writes to the filesystem

2020-01-16 Thread Anindya Haldar
Any take on this very specific question? Sincerely, Anindya Haldar Oracle Responsys > On Jan 15, 2020, at 1:59 PM, Anindya Haldar wrote: > > Okay, let’s say > > - the application is using a non-transactional producer, shared across > multiple threads > - the linger.ms and buffer.memory is no

Re: Kafka send() call, ack, and message writes to the filesystem

2020-01-15 Thread Anindya Haldar
Okay, let’s say - the application is using a non-transactional producer, shared across multiple threads - the linger.ms and buffer.memory is non-zero, and so is batch.size such that messages are actually batched - the replication factor is 3 - the minimum number of ISRs is 2 - the parameter ack

Re: Kafka send() call, ack, and message writes to the filesystem

2020-01-15 Thread M. Manna
Hey Anindya, On Wed, 15 Jan 2020 at 18:23, Anindya Haldar wrote: > Thanks for the response. > > Essentially, we are looking for a confirmation that a send acknowledgement > received at the client’s end will ensure the message is indeed persisted to > the replication logs. We initially wondered

Re: Kafka send() call, ack, and message writes to the filesystem

2020-01-15 Thread Anindya Haldar
Thanks for the response. Essentially, we are looking for a confirmation that a send acknowledgement received at the client’s end will ensure the message is indeed persisted to the replication logs. We initially wondered whether the client has to make an explicit flush() call or whether it has t

Re: Kafka send() call, ack, and message writes to the filesystem

2020-01-15 Thread M. Manna
Anindya, On Wed, 15 Jan 2020 at 16:49, Anindya Haldar wrote: > In our case, the minimum in-sync replicas is set to 2. > > Given that, what will be expected behavior for the scenario I outlined? > This means you will get confirmation when 2 of them have acknowledged. so you will always have 2 i

Re: Kafka send() call, ack, and message writes to the filesystem

2020-01-15 Thread Anindya Haldar
In our case, the minimum in-sync replicas is set to 2. Given that, what will be expected behavior for the scenario I outlined? Sincerely, Anindya Haldar Oracle Responsys > On Jan 15, 2020, at 6:38 AM, Ismael Juma wrote: > > To all the in-sync replicas. You can set the minimum number of in-syn

Re: Kafka send() call, ack, and message writes to the filesystem

2020-01-15 Thread Ismael Juma
To all the in-sync replicas. You can set the minimum number of in-sync replicas via the min.insync.replicas topic/broker config. Ismael On Tue, Jan 14, 2020 at 11:11 AM Anindya Haldar wrote: > I have a question related to the semantics of a producer send and the get > calls on the future return