> if we are not using EBS In that case, what's the point of keeping the broker id? The data will be lost anyway right?
On Thu, Nov 15, 2018, 11:40 AM Srinivas Rapolu <cnu.t...@gmail.com> wrote: > Yes, I understand we need to specify the required broker.id in > server.properties/meta.properties file in-order to retain the broker.id > which was failed. > > We can write custom script to query the zookeeper on launch process to get > the broker.id needs to be set. > > Do we have any standards/script/way defined/preferred for doing this or > suggested by Kafka experts if we are not using EBS? > > Thanks and Regards, > Srinivas > > On Thu, Nov 15, 2018 at 7:10 AM Kaufman Ng <kauf...@confluent.io> wrote: > > > Srinivas, > > > > You might want to check out the AWS best practices from this blog post: > > > > > https://www.confluent.io/blog/design-and-deployment-considerations-for-deploying-apache-kafka-on-aws/ > > > > Kafka broker ids by default are auto-generated, but you can also specify > > values (in server.properties file). By having static broker ids, it's > > easier to recover in general. The blog post above mentions it in more > > details. > > > > Good luck. > > > > On Thu, Nov 15, 2018 at 8:09 AM amit pal <amit5...@gmail.com> wrote: > > > > > If at all you were hell bent on doing this, you could use zookeeper to > > find > > > out the health of current brokers along with their broker id. That > should > > > help re-spin/start the unhealthy instance with same instance id. > > > > > > On Thu, Nov 15, 2018 at 6:08 PM Srinivas Rapolu <cnu.t...@gmail.com> > > > wrote: > > > > > > > having all this stored in DB is getting too complicated, especially > > with > > > > instance level storage and not EBS. > > > > > > > > I am sure there should be easy way to retain the old broker.id for > new > > > AWS > > > > instance spun-up for auto replication. > > > > > > > > Any other ideas/help is appreciated. > > > > > > > > On Thu, Nov 15, 2018 at 2:27 AM Eno Thereska <eno.there...@gmail.com > > > > > > wrote: > > > > > > > > > The general answer depends on what control plane software is taking > > > care > > > > of > > > > > your Kafka deployment. You probably have a layer that launches > Kafka > > > > > instances and monitors their health, right? If so, that layer > should > > > take > > > > > care of the mapping between instances and broker IDs and keep that > > in a > > > > > table persisted somewhere (e.g., DynamoDB). > > > > > > > > > > Eno > > > > > > > > > > On Wed, Nov 14, 2018 at 7:38 PM Srinivas Rapolu < > cnu.t...@gmail.com> > > > > > wrote: > > > > > > > > > > > EBS is one of the option. But we use instance level storage where > > we > > > > > loose > > > > > > all data as soon as we have a broker failed in AWS. > > > > > > > > > > > > In such scenario, anyone has better launch script or cofiguration > > can > > > > be > > > > > > executed on new broker to retain the old id not conflicting with > > > > existing > > > > > > broker ids. > > > > > > > > > > > > On Wed, Nov 14, 2018, 11:58 AM Andrey Dyachkov < > > > > > andrey.dyach...@gmail.com > > > > > > wrote: > > > > > > > > > > > > > You can attach EBS volume, which will store data and > > metadata(e.g. > > > > > broker > > > > > > > id), and then attach it to the new AWS instance and start > Kafka, > > it > > > > > will > > > > > > > pick the broker id plus you won’t need to rebalance the > cluster. > > > > > > > > > > > > > > On Wed 14. Nov 2018 at 19:48, naresh Goud < > > > > nareshgoud.du...@gmail.com> > > > > > > > wrote: > > > > > > > > > > > > > > > Static IP. Buying static IP may help. I am not aws expert > > > > > > > > > > > > > > > > On Wed, Nov 14, 2018 at 12:47 PM Srinivas Rapolu < > > > > cnu.t...@gmail.com > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > Hello Kafka experts, > > > > > > > > > > > > > > > > > > We are running Kafka on AWS, main question is what is the > > best > > > > way > > > > > to > > > > > > > > > retain broker.id on new instance spun-up in-place of > > > > > instance/broker > > > > > > > > > failed. > > > > > > > > > > > > > > > > > > We are currently running Kafka in AWS with broker.id gets > > auto > > > > > > > > generated. > > > > > > > > > But we are having issues when a broker is failed, new > > > > > broker/instance > > > > > > > > > spun-up in AWS get assigned with new broker.id. The issue > > is, > > > > with > > > > > > > this > > > > > > > > > approach, we need to re-assign the topics/replications on > to > > > the > > > > > new > > > > > > > > broker > > > > > > > > > manually. > > > > > > > > > > > > > > > > > > We learned that, replication can be auto resolved by Kafka, > > if > > > we > > > > > can > > > > > > > > > manage to get the same broker.id on the new AWS instance > > > spun-up > > > > > > > > in-place > > > > > > > > > of failed broker/instance. > > > > > > > > > > > > > > > > > > I have read, we can set broker.id.generation.enable= false, > > but > > > > > what > > > > > > is > > > > > > > > the > > > > > > > > > best way to identify and retain the broker.id? Any > > links/help > > > is > > > > > > > > > appreciated. > > > > > > > > > Thanks and Regards, > > > > > > > > > Cnu > > > > > > > > > > > > > > > > > -- > > > > > > > > Thanks, > > > > > > > > Naresh > > > > > > > > www.linkedin.com/in/naresh-dulam > > > > > > > > http://hadoopandspark.blogspot.com/ > > > > > > > > > > > > > > > -- > > > > > > > Thanks, > > > > > > > Andrey > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > Thanks and Regards, > > > Amit Pal > > > > > > > > > -- > > Kaufman Ng > > +1 646 961 8063 > > Solutions Architect | Confluent | www.confluent.io > > >
