Any thoughts on adding a "broker.id.generation.enabled" configuration for auto generating broker ids?
If we added this and defaulted it to false, users who upgrade don't need to worry about the reserved.broker.max.id configuration since it will not be used. Users using the standard/old way of manually configuring ids can keep doing so without concern. And when users enable broker id generation, the docs can inform them how correctly set the reserved.broker.max.id. On Fri, Jan 8, 2016 at 4:18 PM, Grant Henke <ghe...@cloudera.com> wrote: > I agree that many people id their brokers differently and increasing the > default will only handle a subset of those schemes. Though I think > increasing it to some reasonable value may help decrease issues drastically > regardless. > > I also think some longer term fix that avoids collisions all together > would be nice. Though I am not sure what that long term solution is. We > would need to introduce something that a configured broker id is not > allowed to set. Any ideas? > > I also wanted to note here that while investigating this I found some > interesting special cases/rules for the reserved.broker.max.id config. > > 1. Because a zookeeper sequence value is added to that value to generate > the unique id's, the value once configured and used *cannot be decreased* > and still guaranteeing no collisions. > > 2. Because the id was generated it can never be manually set in the > config. Therefore if you need to stand up a new machine with the same > broker ids (perhaps for recovery) you can't set this value manually. The > workaround would be to set the value in the meta.properties file of all the > log directories. (note: I haven't fully vetted this yet) > > > > > On Wed, Dec 23, 2015 at 5:25 PM, Ewen Cheslack-Postava <e...@confluent.io> > wrote: > >> Which other numbering schemes do we want to be able to un-break by >> increasing this default? For example, I know some people use the IP >> address >> with dots removed -- we'd have to use a very large # to make sure that >> worked. Before making another change, it'd be good to know what other >> schemes people are using and that we'd really be fixing the issue for >> someone. >> >> -Ewen >> >> On Fri, Dec 18, 2015 at 9:37 AM, Ismael Juma <ism...@juma.me.uk> wrote: >> >> > On Fri, Dec 18, 2015 at 4:44 PM, Grant Henke <ghe...@cloudera.com> >> wrote: >> > >> > > There is some discussion on KAFKA-1070 >> > > <https://issues.apache.org/jira/browse/KAFKA-1070> around the design >> > > choice >> > > and compatibility. The value 1000 was thrown out as a quick example >> but >> > it >> > > was never discussed beyond that. The discussion also sites a few cases >> > > where a value of 1000 would cause issue. >> > > >> > >> > Thanks for digging that up. Also worth noting that Jay said: >> > >> > "I think we can get around the problem you point out by just defaulting >> the >> > node id sequence to 1000. This could theoretically conflict but most >> people >> > number from 0 or 1 and we can discuss this in the release notes. Our >> plan >> > will be to release with support for both configured node ids and >> assigned >> > node ids for compatibility. After a couple of releases we will remove >> the >> > config." >> > >> > Ismael >> > >> >> >> >> -- >> Thanks, >> Ewen >> > > > > -- > Grant Henke > Software Engineer | Cloudera > gr...@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke > -- Grant Henke Software Engineer | Cloudera gr...@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke
