Hi 22.08.2016, 19:16, "Tom Crayford" <tcrayf...@heroku.com>: > Hi, > > I don't think I understand *why* you need this. Kafka is by default a > distributed HA system that balances data and leadership over nodes. Why do > you need to change this?
In fact, kafka does not balance anything. It uses static partition distribution. In most cases the first replica in replica list is always a leader. If a replica is under heavy load (for example hard disk was replaced or so on) it is a bad idea to do automatic leader "rebalance". > > You could accomplish something like this with mirror maker, that may make > more sense. > > Thanks > > Tom Crayford > > Heroku Kafka > > On Mon, Aug 22, 2016 at 4:05 PM, Jason Aliyetti <j.aliye...@gmail.com> > wrote: > >> I have a use case that requires a 2 node deployment with a Kafka-backed >> service with the following constraints: >> >> - All data must be persisted to node 1. If node 1 fails (regardless of the >> status of node 2), then the system must stop. >> - If node 2 is up, then it must stay in synch with node 1. >> - If node 2 fails, then service must not be disrupted, but as soon as it >> comes back up and rejoins ISR it must stay in synch. >> >> The deployment is basically a primary node and a cold node with real time >> replication, but no failover to the cold node. >> >> To achieve this I am considering adding a broker-level configuration option >> that would prevent a broker from becoming a leader for any topic partition >> it hosts - this would allow me to enforce that the cold node never take >> leadership for any topics. In conjunction with manipulating a topic's >> "min.insync.replicas" setting at runtime, I should be able to achieve the >> behavior desired (2 if both brokers up, 1 if the standby goes down). >> >> I know this sounds like an edgy use case, but does this sound like a >> reasonable approach? Are there any valid use cases around such a broker or >> topic level configuration (i.e. does this sound like a feature that would >> make sense to open a KIP against)?
