Hi, thanks, very interesting KIP ... I haven't fully digested it yet. We have many users who choose not to use the Java client, so I have concerns about the added complexity in developing the clients. A few questions.
1 - is mixing transactional and non transactional messages on the *same topic-partition* really a requirement ? What use case does it satisfy? 2 - I guess some clients may only be interested to implement the producer idempotency. It's not clear how they could be implemented without having to add the transaction capabilities. As others on this list have said, I too would like to see idempotency as a more basic feature, on top which txns can be built. 3 - The KIP seems focused on a use case where consumption from a topic and subsequent production are part of the producer transaction. It'd be great to see a way to extend the producer transaction to include additional transactional resources, so that the consumption from another topic just becomes a special case of a more general "distributed" txn. Edo -------------------------------------------------- Edoardo Comar IBM MessageHub eco...@uk.ibm.com IBM UK Ltd, Hursley Park, SO21 2JN IBM United Kingdom Limited Registered in England and Wales with number 741598 Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU From: Guozhang Wang <wangg...@gmail.com> To: "dev@kafka.apache.org" <dev@kafka.apache.org> Date: 30/11/2016 22:20 Subject: [DISCUSS] KIP-98: Exactly Once Delivery and Transactional Messaging Hi all, I have just created KIP-98 to enhance Kafka with exactly once delivery semantics: *https://cwiki.apache.org/confluence/display/KAFKA/KIP-98+-+Exactly+Once+Delivery+and+Transactional+Messaging < https://cwiki.apache.org/confluence/display/KAFKA/KIP-98+-+Exactly+Once+Delivery+and+Transactional+Messaging >* This KIP adds a transactional messaging mechanism along with an idempotent producer implementation to make sure that 1) duplicated messages sent from the same identified producer can be detected on the broker side, and 2) a group of messages sent within a transaction will atomically be either reflected and fetchable to consumers or not as a whole. The above wiki page provides a high-level view of the proposed changes as well as summarized guarantees. Initial draft of the detailed implementation design is described in this Google doc: https://docs.google.com/document/d/11Jqy_GjUGtdXJK94XGsEIK7CP1SnQGdp2eF 0wSw9ra8 We would love to hear your comments and suggestions. Thanks, -- Guozhang Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
