You need to persist which producer is the leader so that when a new broker takes over it can find out. There could be a special fencing topic that never deleted messages ( using log compaction to save space ). You'd need to think about all the edge cases and race conditions.
Dave > On Apr 29, 2016, at 03:32, Enrico Olivelli - Diennea > <enrico.olive...@diennea.com> wrote: > > Hi, > I would like to use Kafka as transaction log in order to support a case of > replicated state machine, but actually (using 0.9.x) there is a feature I > would like to have. > I'm using Apache BookKeeper and this feature (fencing) is native, but I have > some cases of customers which already use Kafka and I would like to let them > use Kafka as log manager. > > My scenario is the following: > - I have a group of processes which compete to acquire leadership (for > instance using ZooKeeper facilities). > - The only one process which is the leader can write to a shared commit log > (a topic with a single partition) > - The other processes follow the leader, "tailing" the commit log and > replaying the log to the local state > > When the leader loses its role, for instance ZK session expires, I want the > "log manager" to prevent it from writing new entries (fencing). > This action will guarantee that only one service can really "act" as leader > in the group, that is to decide changes to the global state of the system. > > I do not known Kafka internals but I think that a "producer fencing" feature > would be simple to implement if I use a single topic with a single partition. > > As there is only one leader Broker per partition, this leader could enforce > the constraint that only any producer can be active at a time on the > partition. > > Steps: > - Producer A (leader) logs in and writes entries > - Producer B (new leader) logs in, writes a message with a flag "close other > producers for this partition" > - Producer A gets a "ProducerFencedException" and known that someone else > became the new leader > > That do you think? > Is it interesting for others ? > Is there any way to achive this goal using actual Kafka features ? > > Thank you > > > -- > Enrico Olivelli > Software Development Manager @Diennea > Tel.: (+39) 0546 066100 - Int. 925 > Viale G.Marconi 30/14 - 48018 Faenza (RA) > > MagNews - E-mail Marketing Solutions > http://www.magnews.it > Diennea - Digital Marketing Solutions > http://www.diennea.com > > > ________________________________ > > Iscriviti alla nostra newsletter per rimanere aggiornato su digital ed email > marketing! http://www.magnews.it/newsletter/ > > The information in this email is confidential and may be legally privileged. > If you are not the intended recipient please notify the sender immediately > and destroy this email. Any unauthorized, direct or indirect, disclosure, > copying, storage, distribution or other use is strictly forbidden. This e-mail and any files transmitted with it are confidential, may contain sensitive information, and are intended solely for the use of the individual or entity to whom they are addressed. If you have received this e-mail in error, please notify the sender by reply e-mail immediately and destroy all copies of the e-mail and any attachments.
