Hey Andrew, Yeah I think the current state is that we did several design and prototypes (both the transaction work and the idempotence design and the conditional write KIP), but none of these offshoots is really fully rationalized with the other ones. Slow progress in this area has been mainly due to time constraints--no one working on it full time. We're interested in picking up the work at Confluent and hope to get some design ideas out on wiki for discussion in the next month or so.
-Jay On Sun, Feb 21, 2016 at 5:52 AM, Andrew Schofield < andrew_schofield_j...@outlook.com> wrote: > Hi, > That's good, but somewhat mysterious :) I'd like to help accelerate > exactly-one behaviour. Perhaps it might be a candidate for the release > after 0.10, if there's sufficient interest in the community. > > Could someone share the latest thinking on this subject? I'm not sure what > the appropriate forum is. How have you previously been sharing high-level > architectural plans for people to pick up and deliver? Since there's no KIP > I guess it's a bit premature to discuss on the KIP call. > > Thanks > Andrew > > > Subject: Re: Exactly-once publication behaviour > > From: b...@confluent.io > > Date: Fri, 19 Feb 2016 15:13:07 -0800 > > To: users@kafka.apache.org > > > > Hi Andrew > > > > There are plans to add exactly once behaviour. This will likely be a > little more than Idempotent producers with the motivation being to provide > better delivery guarantees for Connect, Streams and Mirror Maker. > > > > B > > > > > > > >> On 19 Feb 2016, at 13:54, Andrew Schofield wrote: > >> > >> When publishing messages to Kafka, you make a choice between > at-most-once and at-least-once delivery, depending on whether you wait for > acknowledgments and whether you retry on failures. In most cases, those > options are good enough. However, some systems offer exactly-once > reliability too. Although my view is that the practical use of exactly-once > is limited in the situations that Kafka is generally used for, when you're > connecting other systems to Kafka or bridging between protocols, I think > there is value in propagating the reliability level that the other system > expects. > >> > >> As a consumer, you can manage your offset and get exactly-once > delivery, or more likely exactly-once processing, of the messages. > >> > >> I've read about idempotent producers ( > https://cwiki.apache.org/confluence/display/KAFKA/Idempotent+Producer) > and I know there's been some discussion about transactions too. > >> > >> Is there a plan to provide the tools to enable exactly-once publication > behaviour? Is this a planned enhancement to Kafka Connect? Is there already > some technique that people are using effectively to get exactly-once? > >> > >> Andrew Schofield > > > >
