Hi Andrew There are plans to add exactly once behaviour. This will likely be a little more than Idempotent producers with the motivation being to provide better delivery guarantees for Connect, Streams and Mirror Maker.
B > On 19 Feb 2016, at 13:54, Andrew Schofield > <andrew_schofield_j...@outlook.com> wrote: > > When publishing messages to Kafka, you make a choice between at-most-once and > at-least-once delivery, depending on whether you wait for acknowledgments and > whether you retry on failures. In most cases, those options are good enough. > However, some systems offer exactly-once reliability too. Although my view is > that the practical use of exactly-once is limited in the situations that > Kafka is generally used for, when you're connecting other systems to Kafka or > bridging between protocols, I think there is value in propagating the > reliability level that the other system expects. > > As a consumer, you can manage your offset and get exactly-once delivery, or > more likely exactly-once processing, of the messages. > > I've read about idempotent producers > (https://cwiki.apache.org/confluence/display/KAFKA/Idempotent+Producer) and I > know there's been some discussion about transactions too. > > Is there a plan to provide the tools to enable exactly-once publication > behaviour? Is this a planned enhancement to Kafka Connect? Is there already > some technique that people are using effectively to get exactly-once? > > Andrew Schofield
