That makes sense. It will be worth starting a discussion and a wiki on a generally applicable control message format. Would you like to file a JIRA for that?
Thanks, Neha On Wed, Sep 10, 2014 at 1:40 PM, Chris Riccomini < criccom...@linkedin.com.invalid> wrote: > Hey Guys, > > The current transactionality proposal ( > https://cwiki.apache.org/confluence/display/KAFKA/Transactional+Messaging+in+Kafka) > and implementation use control messages to handle transactions in Kafka. > Kafka traditionally hasn't had control messages in its topics. > Transactionality (as it's implemented) introduces this pattern, but appears > to do so in a very specific fashion (control messages only for > transactions). > > It seems to me that a good approach to control messages would be to > generalize the control message model in Kafka to support not just > transaction control messages, but arbitrary control messages? On the > producer side, arbitrary control messages should be allowed to be sent, and > on the consumer side, these control messages should be dropped by default. > > Just like transactionality, this would let frameworks (e.g. Samza) and > other app-specific implementations take advantage of in-topic control > messages (as opposed to out of band control messages) without any impact on > existing consumers. > > Thoughts? > > Cheers, > Chris >