You should definitely use the same id if you still have the data - it makes life so much better.
There are 3 common ways to do it: 1. Use the last 3 digits of the IP as the broker ID (assuming Docker gives you the same IP when the container relaunches) 2. Use a deployment manager that can register the brokers in an external DB to make sure the same broker always gets the same ID 3. Use Kafka 0.9.0 where Kafka can auto-generate the broker name and then store it in a data file, so if you still have the files, you have the same broker ID On Tue, Nov 24, 2015 at 2:58 PM, SpikyHawk SpikyHawk < listas.luaf...@gmail.com> wrote: > Hi there. > > I have a similar question but is related to this scenario. > > Docker server running in EC2 instance with an EBS volume attached to it. > > Kafka running in a Docker container in this host with server.properties > "autogenerated" using a bootstrap script. > Part of the bootstrapping process is to generate a broker id using some > property of the host like ip or of the docker container like docker > hostname and update server.properties > > Docker container running Kafka mounts the host attached EBS volume to use > it as log dir. > > case 1: > If container dies and is re-launched on same host, then it will generate a > new broker id but will "see" the same log dir, so data in this dir was > technically generated by a broker with a different id. is this the same > scenario like trying to rename a broker id? Should I try to use same broker > id if this broker will use same log dir with data in it? > > case 2: > If entire host dies or crashes or is terminated by AWS, then can I use the > same EBS with another host where a new kafka container will run with a > different broker id? I think at the end this is same scenario than case 1 > :) > > How do you handle this situation where a broker or host may disappear but > log data is still there? Should always be discarded along with the broker > (id) that generated it? > > Regards > Luciano > > > On Fri, Nov 6, 2015 at 7:24 PM, Todd Palino <tpal...@gmail.com> wrote: > > > I’m not quite sure why you would need to do this - the broker IDs are not > > significant outside of the internal metadata. But this is what you would > > have to do for each move (assuming you are running with at least > > replication factor 2): > > > > 1) Shut down the broker > > 2) Clear its partition data > > 3) Reconfigure the broker to the new ID > > 4) Restart the broker > > 5) Issue a partition reassignment to reassign all of broker 1’s > partitions > > to broker 0 > > 6) Wait for the broker to replicate all it’s partitions from other > members > > of the cluster > > > > That’s a lot of moving data around, just to renumber. You can’t just > issue > > the reassignment while the broker is down, and not delete the partitions, > > because the ID number 0 is unknown, so the reassignment will fail (the > > broker is not online). If you wanted to shut the entire cluster down you > > could, in theory, walk through the Zookeeper tree manually changing all > the > > replica information. That assumes you can shut the whole cluster down > for a > > while. > > > > -Todd > > > > > > > > On Fri, Nov 6, 2015 at 1:23 PM, Arathi Maddula <amadd...@boardreader.com > > > > wrote: > > > > > Hi, > > > Is it possible to change the broker.ids property for a node belonging > to > > a > > > Kafka cluster? For example, currently if I have brokers with ids > 1,2,3. > > If > > > I want to stop broker 1, can I change broker.id to 0 (with current > id = > > > 1) in server.properties and meta.properties files and then restart > broker > > > 1. Can I repeat this for brokers 2 and 3 as well? > > > > > > Thanks, > > > Arathi > > > > > > > > >
