Hi Jay, Thanks for the response. Happy to engage in the discussions once there's something concrete on the wiki.
Andrew ---------------------------------------- > Date: Sun, 21 Feb 2016 10:23:55 -0800 > Subject: Re: Exactly-once publication behaviour > From: j...@confluent.io > To: users@kafka.apache.org > > Hey Andrew, > > Yeah I think the current state is that we did several design and prototypes > (both the transaction work and the idempotence design and the conditional > write KIP), but none of these offshoots is really fully rationalized with > the other ones. Slow progress in this area has been mainly due to time > constraints--no one working on it full time. We're interested in picking up > the work at Confluent and hope to get some design ideas out on wiki for > discussion in the next month or so. > > -Jay > > On Sun, Feb 21, 2016 at 5:52 AM, Andrew Schofield < > andrew_schofield_j...@outlook.com> wrote: > >> Hi, >> That's good, but somewhat mysterious :) I'd like to help accelerate >> exactly-one behaviour. Perhaps it might be a candidate for the release >> after 0.10, if there's sufficient interest in the community. >> >> Could someone share the latest thinking on this subject? I'm not sure what >> the appropriate forum is. How have you previously been sharing high-level >> architectural plans for people to pick up and deliver? Since there's no KIP >> I guess it's a bit premature to discuss on the KIP call. >> >> Thanks >> Andrew >> >>> Subject: Re: Exactly-once publication behaviour >>> From: b...@confluent.io >>> Date: Fri, 19 Feb 2016 15:13:07 -0800 >>> To: users@kafka.apache.org >>> >>> Hi Andrew >>> >>> There are plans to add exactly once behaviour. This will likely be a >> little more than Idempotent producers with the motivation being to provide >> better delivery guarantees for Connect, Streams and Mirror Maker. >>> >>> B >>> >>> >>> >>>> On 19 Feb 2016, at 13:54, Andrew Schofield wrote: >>>> >>>> When publishing messages to Kafka, you make a choice between >> at-most-once and at-least-once delivery, depending on whether you wait for >> acknowledgments and whether you retry on failures. In most cases, those >> options are good enough. However, some systems offer exactly-once >> reliability too. Although my view is that the practical use of exactly-once >> is limited in the situations that Kafka is generally used for, when you're >> connecting other systems to Kafka or bridging between protocols, I think >> there is value in propagating the reliability level that the other system >> expects. >>>> >>>> As a consumer, you can manage your offset and get exactly-once >> delivery, or more likely exactly-once processing, of the messages. >>>> >>>> I've read about idempotent producers ( >> https://cwiki.apache.org/confluence/display/KAFKA/Idempotent+Producer) >> and I know there's been some discussion about transactions too. >>>> >>>> Is there a plan to provide the tools to enable exactly-once publication >> behaviour? Is this a planned enhancement to Kafka Connect? Is there already >> some technique that people are using effectively to get exactly-once? >>>> >>>> Andrew Schofield >>> >> >>
