Hi, I would like to use Kafka as transaction log in order to support a case of replicated state machine, but actually (using 0.9.x) there is a feature I would like to have. I'm using Apache BookKeeper and this feature (fencing) is native, but I have some cases of customers which already use Kafka and I would like to let them use Kafka as log manager.
My scenario is the following: - I have a group of processes which compete to acquire leadership (for instance using ZooKeeper facilities). - The only one process which is the leader can write to a shared commit log (a topic with a single partition) - The other processes follow the leader, "tailing" the commit log and replaying the log to the local state When the leader loses its role, for instance ZK session expires, I want the "log manager" to prevent it from writing new entries (fencing). This action will guarantee that only one service can really "act" as leader in the group, that is to decide changes to the global state of the system. I do not known Kafka internals but I think that a "producer fencing" feature would be simple to implement if I use a single topic with a single partition. As there is only one leader Broker per partition, this leader could enforce the constraint that only any producer can be active at a time on the partition. Steps: - Producer A (leader) logs in and writes entries - Producer B (new leader) logs in, writes a message with a flag "close other producers for this partition" - Producer A gets a "ProducerFencedException" and known that someone else became the new leader That do you think? Is it interesting for others ? Is there any way to achive this goal using actual Kafka features ? Thank you -- Enrico Olivelli Software Development Manager @Diennea Tel.: (+39) 0546 066100 - Int. 925 Viale G.Marconi 30/14 - 48018 Faenza (RA) MagNews - E-mail Marketing Solutions http://www.magnews.it Diennea - Digital Marketing Solutions http://www.diennea.com ________________________________ Iscriviti alla nostra newsletter per rimanere aggiornato su digital ed email marketing! http://www.magnews.it/newsletter/ The information in this email is confidential and may be legally privileged. If you are not the intended recipient please notify the sender immediately and destroy this email. Any unauthorized, direct or indirect, disclosure, copying, storage, distribution or other use is strictly forbidden.
