Yes, sorry, I think this is right -- it's pretty application-specific. One thing to note: in a large subset of cases (ie. bulk load, copycat-type, mirrormaker) the correct response is not to resend the message at all; if there's already a message at that offset, it's because another instance of the same process already sent the exact same message.
On Mon, Jul 20, 2015 at 4:38 PM, Flavio P JUNQUEIRA <f...@apache.org> wrote: > Up to Ben to clarify, but I'd think that in this case, it is up to the > logic of B to decide what to do. B knows that the offset isn't what it > expects, so it can react accordingly. If it chooses to try again, then it > should not violate any application invariant. > > -Flavio > > On Fri, Jul 17, 2015 at 9:49 PM, Ashish Singh <asi...@cloudera.com> wrote: > >> Good concept. I have a question though. >> >> Say there are two producers A and B. Both producers are producing to same >> partition. >> - A sends a message with expected offset, x1 >> - Broker accepts is and sends an Ack >> - B sends a message with expected offset, x1 >> - Broker rejects it, sends nack >> - B sends message again with expected offset, x1+1 >> - Broker accepts it and sends Ack >> I guess this is what this KIP suggests, right? If yes, then how does this >> ensure that same message will not be written twice when two producers are >> producing to same partition? Producer on receiving a nack will try again >> with next offset and will keep doing so till the message is accepted. Am I >> missing something? >> >> Also, you have mentioned on KIP, "it imposes little to no runtime cost in >> memory or time", I think that is not true for time. With this approach >> producers' performance will reduce proportionally to number of producers >> writing to same partition. Please correct me if I am missing out something. >> >> >> On Fri, Jul 17, 2015 at 11:32 AM, Mayuresh Gharat < >> gharatmayures...@gmail.com> wrote: >> >> > If we have 2 producers producing to a partition, they can be out of >> order, >> > then how does one producer know what offset to expect as it does not >> > interact with other producer? >> > >> > Can you give an example flow that explains how it works with single >> > producer and with multiple producers? >> > >> > >> > Thanks, >> > >> > Mayuresh >> > >> > On Fri, Jul 17, 2015 at 10:28 AM, Flavio Junqueira < >> > fpjunque...@yahoo.com.invalid> wrote: >> > >> > > I like this feature, it reminds me of conditional updates in zookeeper. >> > > I'm not sure if it'd be best to have some mechanism for fencing rather >> > than >> > > a conditional write like you're proposing. The reason I'm saying this >> is >> > > that the conditional write applies to requests individually, while it >> > > sounds like you want to make sure that there is a single client writing >> > so >> > > over multiple requests. >> > > >> > > -Flavio >> > > >> > > > On 17 Jul 2015, at 07:30, Ben Kirwin <b...@kirw.in> wrote: >> > > > >> > > > Hi there, >> > > > >> > > > I just added a KIP for a 'conditional publish' operation: a simple >> > > > CAS-like mechanism for the Kafka producer. The wiki page is here: >> > > > >> > > > >> > > >> > >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-27+-+Conditional+Publish >> > > > >> > > > And there's some previous discussion on the ticket and the users >> list: >> > > > >> > > > https://issues.apache.org/jira/browse/KAFKA-2260 >> > > > >> > > > >> > > >> > >> https://mail-archives.apache.org/mod_mbox/kafka-users/201506.mbox/%3CCAAeOB6ccyAA13YNPqVQv2o-mT5r=c9v7a+55sf2wp93qg7+...@mail.gmail.com%3E >> > > > >> > > > As always, comments and suggestions are very welcome. >> > > > >> > > > Thanks, >> > > > Ben >> > > >> > > >> > >> > >> > -- >> > -Regards, >> > Mayuresh R. Gharat >> > (862) 250-7125 >> > >> >> >> >> -- >> >> Regards, >> Ashish >>
