Hello all, I updated the KIP document with the changes discussed here. In particular:
1. The consumer config was renamed from 'fetch.mode' to 'isolation.level'. The values were changed appropriately as well. 2. Clarified the transactional guarantees for the consumer. 3. Added a subsection about the streams usecase for transactions, as this is a driving motivation behind the current proposal. The existing motivation was not strong enough, as evidenced by some of the discussions that took place here. 4. Miscellaneous minor clarifications which have been pointed out in the thread by multiple folks. I have not yet updated the 'Rejected alternatives' since we have a bunch of higher level proposals which are a bit open right now. I think the proposals bifurcate into doing buffering client side vs complicating the server side when handling transactions. We plan on doing a KIP call this month where we can discuss our options in this regard, at which point we will update the document (and rejected alternatives) to reflect the collective decision. Thanks for all the comments, it has been a great discussion so far! Here is the KIP link, for convenience: https://cwiki.apache.org/confluence/display/KAFKA/KIP-98+-+Exactly+Once+Delivery+and+Transactional+Messaging Apurva On Wed, Nov 30, 2016 at 2:19 PM, Guozhang Wang <wangg...@gmail.com> wrote: > Hi all, > > I have just created KIP-98 to enhance Kafka with exactly once delivery > semantics: > > *https://cwiki.apache.org/confluence/display/KAFKA/KIP- > 98+-+Exactly+Once+Delivery+and+Transactional+Messaging > <https://cwiki.apache.org/confluence/display/KAFKA/KIP- > 98+-+Exactly+Once+Delivery+and+Transactional+Messaging>* > > This KIP adds a transactional messaging mechanism along with an idempotent > producer implementation to make sure that 1) duplicated messages sent from > the same identified producer can be detected on the broker side, and 2) a > group of messages sent within a transaction will atomically be either > reflected and fetchable to consumers or not as a whole. > > The above wiki page provides a high-level view of the proposed changes as > well as summarized guarantees. Initial draft of the detailed implementation > design is described in this Google doc: > > https://docs.google.com/document/d/11Jqy_GjUGtdXJK94XGsEIK7CP1SnQGdp2eF > 0wSw9ra8 > > > We would love to hear your comments and suggestions. > > Thanks, > > -- Guozhang >
