The key feature of BookKeeper in this case is that it guarantees that exactly only one "leader" can change the state of the machine, you can think about it as a way of implementing a consensus algorithm.
For instance if the "old" leader (Kafka producer in this case) succeeds to write to the log, it will update its local state, concurrently another leader succeeds to write to the same topic. It will be possible for the followers to detect this "new leadership" but is it not possible for the old "leader" to detect this case until it detects to be partitioned away from the group (using the external leader election system) and thus it will have a diverged view of the state of the global system. IMHO A system which implements only the leader election and does not leverage such a fencing feature cannot implement correctly a distributed state replication using a distributed log. Thanks -- Enrico Il giorno ven, 29/04/2016 alle 11.55 +0000, Tauzell, Dave ha scritto: You need to persist which producer is the leader so that when a new broker takes over it can find out. There could be a special fencing topic that never deleted messages ( using log compaction to save space ). You'd need to think about all the edge cases and race conditions. Dave On Apr 29, 2016, at 03:32, Enrico Olivelli - Diennea <enrico.olive...@diennea.com<mailto:enrico.olive...@diennea.com>> wrote: Hi, I would like to use Kafka as transaction log in order to support a case of replicated state machine, but actually (using 0.9.x) there is a feature I would like to have. I'm using Apache BookKeeper and this feature (fencing) is native, but I have some cases of customers which already use Kafka and I would like to let them use Kafka as log manager. My scenario is the following: - I have a group of processes which compete to acquire leadership (for instance using ZooKeeper facilities). - The only one process which is the leader can write to a shared commit log (a topic with a single partition) - The other processes follow the leader, "tailing" the commit log and replaying the log to the local state When the leader loses its role, for instance ZK session expires, I want the "log manager" to prevent it from writing new entries (fencing). This action will guarantee that only one service can really "act" as leader in the group, that is to decide changes to the global state of the system. I do not known Kafka internals but I think that a "producer fencing" feature would be simple to implement if I use a single topic with a single partition. As there is only one leader Broker per partition, this leader could enforce the constraint that only any producer can be active at a time on the partition. Steps: - Producer A (leader) logs in and writes entries - Producer B (new leader) logs in, writes a message with a flag "close other producers for this partition" - Producer A gets a "ProducerFencedException" and known that someone else became the new leader That do you think? Is it interesting for others ? Is there any way to achive this goal using actual Kafka features ? Thank you -- Enrico Olivelli Software Development Manager @Diennea Tel.: (+39) 0546 066100 - Int. 925 Viale G.Marconi 30/14 - 48018 Faenza (RA) MagNews - E-mail Marketing Solutions http://www.magnews.it Diennea - Digital Marketing Solutions http://www.diennea.com ________________________________ Iscriviti alla nostra newsletter per rimanere aggiornato su digital ed email marketing! http://www.magnews.it/newsletter/ The information in this email is confidential and may be legally privileged. If you are not the intended recipient please notify the sender immediately and destroy this email. Any unauthorized, direct or indirect, disclosure, copying, storage, distribution or other use is strictly forbidden. This e-mail and any files transmitted with it are confidential, may contain sensitive information, and are intended solely for the use of the individual or entity to whom they are addressed. If you have received this e-mail in error, please notify the sender by reply e-mail immediately and destroy all copies of the e-mail and any attachments. -- Enrico Olivelli Software Development Manager @Diennea Tel.: (+39) 0546 066100 - Int. 925 Viale G.Marconi 30/14 - 48018 Faenza (RA) MagNews - E-mail Marketing Solutions http://www.magnews.it Diennea - Digital Marketing Solutions http://www.diennea.com ________________________________ Iscriviti alla nostra newsletter per rimanere aggiornato su digital ed email marketing! http://www.magnews.it/newsletter/ The information in this email is confidential and may be legally privileged. If you are not the intended recipient please notify the sender immediately and destroy this email. Any unauthorized, direct or indirect, disclosure, copying, storage, distribution or other use is strictly forbidden.
