On Wed, Feb 27, 2019, at 14:12, Harsha wrote: > Hi Colin, > What we want to is to preserve the broker.id so that we > can do an offline rebuild of a broker. In our cases going through > online Kafka replication to bring up, a failed node will put producer > latencies at risk given the new broker will put all the other leaders > busy with its replication requests. For an offline rebuild, we do not > need to do rebalance as long as we can recover the broker.id > Overall, irrespective of this use case we still want an > ability to retrieve a broker.id for an existing host. This will make > swapping in new hosts with failed hosts by keeping the existing > hostname easier.
Thanks for the explanation. Shouldn't this should be handled by the cluster management tool, though? Kafka doesn't include a mechanism for re-creating nodes that failed. That's up to kubernetes, or ansible, or whatever cluster provisioning framework you have in place. This feels like the same kind of thing: managing how the cluster is provisioned. best, Colin > > Thanks, > Harsha > On Wed, Feb 27, 2019, at 11:53 AM, Colin McCabe wrote: > > Hi Li, > > > > > The mechanism simplifies deployment because the same configuration can > > be > > > used across all brokers, however, in a large system where disk failure > > is > > > a norm, the meta file could often get lost, causing a new broker id > > being > > > allocated. This is problematic because new broker id has no partition > > > assigned to it so it can’t do anything, while partitions assigned to the > > > old one lose one replica > > > > If all of the disks have failed, then the partitions will lose their > > replicas no matter what, right? If any of the disks is still around, > > then there will be a meta file on the disk which contains the previous > > broker ID. So I'm not sure that we need to change anything here. > > > > best, > > Colin > > > > > > On Tue, Feb 5, 2019, at 14:38, Li Kan wrote: > > > Hi, I have KIP-426, which is a small change on automatically determining > > > broker id when starting up. I am new to Kafka so there are a bunch of > > > design trade-offs that I might be missing or hard to decide, so I'd like > > > to > > > get some suggestions on it. I'd expect (and open) to modify (or even > > > totally rewrite) the KIP based on suggestions. Thanks. > > > > > > -- > > > Best, > > > Kan > > > > > >
