Hi Dan, Thank you for the KIP. KIP-226 (https://cwiki.apache.org/ confluence/display/KAFKA/KIP-226+-+Dynamic+Broker+Configuration) proposes to add an option to include all synonyms of a config option when describing configs. This includes any defaults. For example (using Dong's example), if you have topicA with min.cleanable.dirty.ratio=0.6 as an override and the broker default is 0.5, AdminClient#describeConfigs with synonyms would return the actual value in use as the config value (I,e. min.cleanable.dirty.ratio=0.6). And the synonyms list would contain (in the order of precedence in which these configs are applied):
1. min.cleanable.dirty.ratio=0.6, config source=TOPIC_CONFIG 2. log.min.cleanable.dirty.ratio=0.5, config source=STATIC_BROKER_CONFIG KIP-226 doesn't give you exactly what you are proposing in this KIP, but it gives the mapping of configs. My concern with this KIP is that it assumes that if the broker config is static, i.e. if the current value of log.min.cleanable.dirty.ratio=0.6, you can safely create a topic with default min.cleanable.dirty.ratio relying on that the value to be applied all the time. This will not work with dynamic broker configs if the broker defaults can be updated at any time. On Mon, Dec 4, 2017 at 11:22 PM, dan <dan.norw...@gmail.com> wrote: > for point 1 i agree, its not too strong. only addition i could come up with > is that it allows any utility to have better forwards compatability. a cli > written that can inspect how a topic will be created would be able to give > insight/expectations about configs that didn't exist at compilation time. > > for point 2, i am thinking about a situation where the user creating topics > and the user managing brokers are different people with different > skills/abilities. > > so in the given example lets assume a user creating the topic does not > *really* understand what that config means, so they are likely to just > leave it as default. and are happy for their admin to change these on the > broker as needed. > > but say we have another user who is creating a topic who has much more > experience and has done testing, they will be able to determine what the > value is on the cluster and check to see if it matches expectations or > needs to be set. possibly if this is set to something incorrect for their > usecase they will have a reason to verify and speak with the admin about > their usecase. > > > moreover, i think without this added capability it makes it > difficult/impossible to accurately use broker defaults for topics. right > now a user is left to either decide configs a priori and lose this > functionality, or guess/check what they need to set and end in a possibly > bad situation until they can get their *live* topic configured. > > dan > > On Mon, Dec 4, 2017 at 2:50 PM, Dong Lin <lindon...@gmail.com> wrote: > > > Hey Dan, > > > > Thanks again for the update:) > > > > I am not sure I fully understand the points (1) and (2) in the "Always > > Configure ALL Configs For a Topic". In my previous question, I don't mean > > that users should specify full list of topics configs. I mean that user > can > > specify the full list of topic configs he/she wants to override. > > > > Your KIP proposes to allow user to query the default topic configs. In my > > understanding, users want to know the default configs only if he/she > > already has a specific list of (config, value) pairs for which he/she > wants > > to override. The workflow will be that user queries the default configs, > > compares the default value with that specific list, and selectively > > override the configs whose value is different from the default value. > > Therefore, the alternative solution does not suffer the problem (1) > because > > user needs to know that specific list of (config, value) pair anyway. > Does > > this make sense? > > > > Regarding problem (2), I think you are saying that if the user wants to > set > > the min.cleanable.dirty.ratio to be 0.5 and the default value is > currently > > set to 0.5, then user would not want to explicitly override the config > > during topic creation. The purpose is for the min.cleanable.dirty.ratio > of > > this topic to be changed to 0.6 if admin change the default > > min.cleanable.dirty.ratio to 0.6 in the future. > > > > But I am not sure this is a reasonable example. If user wants to > > compare default value of min.cleanable.dirty.ratio with its expected > value > > 0.5, then it seems reasonable for user to override it to be 0.5 during > > topic creation if the default value is currently 0.6. The question is, > why > > would user not want to override the default value to be 0.5 if the > default > > value is changed from 0.5 to 0.6 later? > > > > Thanks, > > Dong > > > > > > On Mon, Dec 4, 2017 at 2:36 PM, dan <dan.norw...@gmail.com> wrote: > > > > > updated again :) > > > > > > by having users always set all configs you lose the ability to use the > > > broker defaults as intended, since topic configs are overlaid. example > in > > > the kip doc. > > > > > > dan > > > > > > On Mon, Dec 4, 2017 at 11:47 AM, Dong Lin <lindon...@gmail.com> wrote: > > > > > > > Hey Dan, > > > > > > > > Thanks for the update. I just want to push the discussion a bit > > further. > > > > Another alternative, which currently is not described in the KIP, is > > for > > > > user to always create the topic with the full list of configs it may > > want > > > > to override. Can you help me understand what is the drawback of this > > > > approach? > > > > > > > > Thanks, > > > > Dong > > > > > > > > On Mon, Dec 4, 2017 at 11:30 AM, dan <dan.norw...@gmail.com> wrote: > > > > > > > > > Dong, > > > > > > > > > > i added a section on current state and workarounds along with my > > > > arguments > > > > > for why they are less than optimal to the wiki. but the jist of it > is > > > you > > > > > can end up with messages in your topic in an incorrect/invalid > state > > if > > > > you > > > > > do this. > > > > > > > > > > thanks, > > > > > dan > > > > > > > > > > On Mon, Dec 4, 2017 at 10:53 AM, Dong Lin <lindon...@gmail.com> > > wrote: > > > > > > > > > > > Hey Dan, > > > > > > > > > > > > Thanks for the KIP. Can you help me understand the motivation by > > > > > providing > > > > > > a use-case that can not be easily completed without this KIP? > > > > > > > > > > > > It seems that most users will simply create the topic without > > > worrying > > > > > > about the default configs. If a user has specific requirement for > > the > > > > > > default configs, he/she can use AdminClient.describeConfigs() and > > > > > > AdminClient.alterConfigs() after the topic has been created. This > > > seems > > > > > to > > > > > > work well. Did I miss something? > > > > > > > > > > > > Thanks, > > > > > > Dong > > > > > > > > > > > > > > > > > > On Mon, Dec 4, 2017 at 9:25 AM, dan <dan.norw...@gmail.com> > wrote: > > > > > > > > > > > > > I would like to start a discussion about KIP-234 > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP- > > > > > > > 234%3A+add+support+for+getting+topic+defaults+from+AdminClient > > > > > > > > > > > > > > thanks > > > > > > > dan > > > > > > > > > > > > > > > > > > > > > > > > > > > >
