2. There was a typo in my previous email. I meant to say that we should use snake case because it's more consistent. I couldn't find any examples of camel case but did find some snake case (jmx_port). Other than that, most other entries are single word keys.
3. The purge frequency is short (15 minutes). So it should be safe to ignore older notifications. Add a story here: https://cwiki.apache.org/confluence/display/KAFKA/KIP-21+-+Dynamic+Configuration#KIP-21-DynamicConfiguration-Migrationplanfornotifications Thanks, Aditya ________________________________________ From: Jay Kreps [jay.kr...@gmail.com] Sent: Saturday, May 30, 2015 9:51 AM To: dev@kafka.apache.org Subject: Re: [VOTE] KIP-21 Dynamic Configuration 1. Great. 2. I don't have a preference as to the casing, but I really appreciate consistency. Is everything using underscores today? If so let's stick with that. If we are already inconsistent then I guess it's too late and we can do whatever. Let me know and I'll update the coding standard. 3. Not sure what the default purge frequency is. I don't think we need to work the details of this out in the KIP, but we need a story for the upgrade path so people don't get bitten. -Jay On Thu, May 28, 2015 at 11:22 PM, Aditya Auradkar < aaurad...@linkedin.com.invalid> wrote: > Yeah, the same cleaning mechanism will be carried over. > > > 1. Are we introducing a new Java API for the config change protocol and > if > > so where will that appear? Is that going to be part of the java api in > the > > admin api kip? Let's document that. > Yeah, we need to introduce a new Java API for the config change protocol. > It should be a part of the AdminClient API. I'll alter KIP-4 to reflect > that since the API is being introduced there. > > > 2. The proposed JSON format uses camel case for field names, is that what > > we've used for other JSON in zookeeper? > I think camel case is more appropriate for the JSON format. For example, > under the "brokers" znode, I found "jmx_port". > > > 3. This changes the format of the notifications, right? How will we > > grandfather in the old format? Clusters will have existing change > > notifications in the old format so I think the new code will need to be > > able to read those? > Interesting, I figured the existing notifications were purged by a cleaner > thread frequently. In that case, we wouldn't need to grandfather any > notifications since we would only need to not make any config changes for X > minutes for there to be no changes in ZK. But the old notifications are > actually removed when a new notification is received or the broker is > bounced. So we do need to handle notifications in the old format. Should we > simply ignore older changes since they are only valid for a short period of > time? > > Thanks, > Aditya > ________________________________________ > From: Jay Kreps [jay.kr...@gmail.com] > Sent: Thursday, May 28, 2015 5:25 PM > To: dev@kafka.apache.org > Subject: Re: [VOTE] KIP-21 Dynamic Configuration > > That is handled now so I am assuming the same mechanism carries over? > > -Jay > > On Thu, May 28, 2015 at 5:12 PM, Guozhang Wang <wangg...@gmail.com> wrote: > > > For the sequential config/changes/config_change_XX znode, do we have any > > manners to do cleaning in order to avoid the change-log from growing > > indefinitely? > > > > Guozhang > > > > On Thu, May 28, 2015 at 5:02 PM, Jay Kreps <jay.kr...@gmail.com> wrote: > > > > > I still have a couple of questions: > > > 1. Are we introducing a new Java API for the config change protocol and > > if > > > so where will that appear? Is that going to be part of the java api in > > the > > > admin api kip? Let's document that. > > > 2. The proposed JSON format uses camel case for field names, is that > what > > > we've used for other JSON in zookeeper? > > > 3. This changes the format of the notifications, right? How will we > > > grandfather in the old format? Clusters will have existing change > > > notifications in the old format so I think the new code will need to be > > > able to read those? > > > > > > -Jay > > > > > > On Thu, May 28, 2015 at 11:41 AM, Aditya Auradkar < > > > aaurad...@linkedin.com.invalid> wrote: > > > > > > > bump > > > > > > > > ________________________________________ > > > > From: Aditya Auradkar > > > > Sent: Tuesday, May 26, 2015 1:16 PM > > > > To: dev@kafka.apache.org > > > > Subject: RE: [VOTE] KIP-21 Dynamic Configuration > > > > > > > > Hey everyone, > > > > > > > > Completed the changes to KIP-4. After today's hangout, there doesn't > > > > appear to be anything remaining to discuss on this KIP. > > > > Please vote so we can formally close this. > > > > > > > > Thanks, > > > > Aditya > > > > > > > > ________________________________________ > > > > From: Aditya Auradkar > > > > Sent: Thursday, May 21, 2015 11:26 AM > > > > To: dev@kafka.apache.org > > > > Subject: RE: [VOTE] KIP-21 Dynamic Configuration > > > > > > > > I think we should remove the config part in TopicMetadataResponse. > It's > > > > probably cleaner if Alter and Describe are the only way to view and > > > modify > > > > configs but I don't feel very strongly about it. > > > > > > > > Re-summarizing the proposed changes to KIP-4: > > > > - Change AlterTopic to not allow setting configs. Config changes will > > > flow > > > > through AlterConfig. CreateTopic will still allow setting configs as > it > > > is > > > > nice to be able to specify configs while creating the topic. > > > > - TopicMetadataResponse shoudn't return config for the topic. > > > > DescribeConfig is the way to go. > > > > - Change "InvalidTopicConfiguration" error code to > > "InvalidEntityConfig" > > > > as proposed in KIP-21. > > > > > > > > Aditya > > > > > > > > ________________________________________ > > > > From: Jun Rao [j...@confluent.io] > > > > Sent: Thursday, May 21, 2015 10:50 AM > > > > To: dev@kafka.apache.org > > > > Subject: Re: [VOTE] KIP-21 Dynamic Configuration > > > > > > > > What about TopicMetadataResponse in KIP-4? Do we remove the config > part > > > in > > > > it? > > > > > > > > Thanks, > > > > > > > > Jun > > > > > > > > On Thu, May 21, 2015 at 10:25 AM, Aditya Auradkar < > > > > aaurad...@linkedin.com.invalid> wrote: > > > > > > > > > Hey Jun, > > > > > > > > > > I've added a section on error codes on the KIP-21 wiki. > > > > > > > > > > Here are the proposed changes to KIP-4. I'll update the wiki > shortly. > > > > > - Change AlterTopic to not allow setting configs. Config changes > will > > > > flow > > > > > through AlterConfig. CreateTopic will still allow setting configs > as > > it > > > > is > > > > > nice to be able to specify configs while creating the topic. > > > > > - Change "InvalidTopicConfiguration" error code to > > > "InvalidEntityConfig" > > > > > as proposed in KIP-21. > > > > > > > > > > > > > > > Thanks, > > > > > Aditya > > > > > > > > > > ________________________________________ > > > > > From: Jun Rao [j...@confluent.io] > > > > > Sent: Thursday, May 21, 2015 8:41 AM > > > > > To: dev@kafka.apache.org > > > > > Subject: Re: [VOTE] KIP-21 Dynamic Configuration > > > > > > > > > > Aditya, > > > > > > > > > > For completeness, could you list the set of error codes in the > wiki? > > > > Also, > > > > > could you summarize the changes that are needed for the requests > > listed > > > > in > > > > > KIP-4 and update the wiki accordingly? > > > > > > > > > > Thanks, > > > > > > > > > > Jun > > > > > > > > > > On Tue, May 19, 2015 at 10:33 PM, Aditya Auradkar < > > > > > aaurad...@linkedin.com.invalid> wrote: > > > > > > > > > > > Thanks Andrii. I'll make the changes. > > > > > > > > > > > > I've also updated KIP-21 to include the new config requests. > Take a > > > > look > > > > > > and vote. > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-21+-+Dynamic+Configuration > > > > > > > > > > > > Aditya > > > > > > ________________________________________ > > > > > > From: Andrii Biletskyi [andrii.bilets...@stealth.ly] > > > > > > Sent: Tuesday, May 19, 2015 2:26 PM > > > > > > To: dev@kafka.apache.org > > > > > > Subject: Re: [VOTE] KIP-21 Dynamic Configuration > > > > > > > > > > > > Hi, > > > > > > > > > > > > Sorry I wasn't able to participate. I don't have objections about > > > > > removing > > > > > > config changes from AlterTopic (as I understand both AddedConfig > > and > > > > > > DeletedConfig) - you are welcome to update the KIP page. > > > > > > > > > > > > Thanks, > > > > > > Andrii Biletskyi > > > > > > > > > > > > On Tue, May 19, 2015 at 11:40 PM, Aditya Auradkar < > > > > > > aaurad...@linkedin.com.invalid> wrote: > > > > > > > > > > > > > Updating the discussion with the latest comments. > > > > > > > > > > > > > > 1. We discussed adding 2 new API's (AlterConfig and > > > DescribeConfig). > > > > > I'll > > > > > > > update KIP-21 with details on these. > > > > > > > 2. Discussed during the KIP hangout. We are in agreement. > > > > > > > > > > > > > > (1) has a dependency on KIP-4 being completed. Rest of the work > > in > > > > the > > > > > > KIP > > > > > > > can be implemented independently. Any concerns if we tackle it > as > > > two > > > > > > > separate work items implementation wise? > > > > > > > > > > > > > > We also discussed changing the AlterTopic command in KIP-4 to > not > > > > > include > > > > > > > config changes. Instead, all config changes will pass through > the > > > > newly > > > > > > > proposed AlterConfig. If no-one objects, I can make some > changes > > to > > > > > KIP-4 > > > > > > > to reflect this. > > > > > > > > > > > > > > Aditya > > > > > > > > > > > > > > ________________________________________ > > > > > > > From: Jay Kreps [jay.kr...@gmail.com] > > > > > > > Sent: Tuesday, May 19, 2015 10:51 AM > > > > > > > To: dev@kafka.apache.org > > > > > > > Subject: Re: [VOTE] KIP-21 Dynamic Configuration > > > > > > > > > > > > > > Hey Aditya, > > > > > > > > > > > > > > Two comments: > > > > > > > > > > > > > > 1. Yeah we need to reconcile this with the APIs in KIP-4. I > think > > > it > > > > > does > > > > > > > make sense to allow setting config during topic creation. I > agree > > > > with > > > > > > your > > > > > > > summary that having alter topic and alter config may be > > confusing, > > > > but > > > > > > > there are also some non-config changes such as replication > factor > > > and > > > > > > > partition count that alter topic can carry out. What is the > final > > > > state > > > > > > you > > > > > > > are proposing? > > > > > > > > > > > > > > 2. This is implementation related so probably can be removed > from > > > the > > > > > KIP > > > > > > > entirely, but you seem to be proposing a separate config > manager > > > for > > > > > each > > > > > > > config override type. Should we just generalize > > TopicConfigManager > > > to > > > > > be > > > > > > > ConfigOverrideManager and have it handle all the override types > > we > > > > will > > > > > > > have? I think I may just be unclear on what you are > proposing... > > > > > > > > > > > > > > -Jay > > > > > > > > > > > > > > On Mon, May 18, 2015 at 1:34 PM, Aditya Auradkar < > > > > > > > aaurad...@linkedin.com.invalid> wrote: > > > > > > > > > > > > > > > Yeah, that was just a typo. I've fixed it. Thanks for calling > > it > > > > out. > > > > > > > > > > > > > > > > In KIP-4, I believe we have 3 types of requests: CreateTopic, > > > > > > AlterTopic > > > > > > > > and DeleteTopic. The topic configs are a sub-type of the > Create > > > and > > > > > > Alter > > > > > > > > commands. I think it would be nice to simply have a > AlterConfig > > > > > command > > > > > > > > that can alter any type of config rather than having a > specific > > > > > > > > ClientConfig. > > > > > > > > > > > > > > > > AlterConfig => [ConfigType [AddedConfigEntry] > [DeletedConfig]] > > > > > > > > ConfigType => string > > > > > > > > AddedConfigEntry => ConfigKey ConfigValue > > > > > > > > ConfigKey => string > > > > > > > > ConfigValue => string > > > > > > > > DeletedConfig => string > > > > > > > > > > > > > > > > The downside of this approach is that we will have 2 separate > > > ways > > > > of > > > > > > > > changing topic configs (AlterTopic and AlterConfig). While a > > > > general > > > > > > > > AlterConfig only makes sense if we plan to have more than two > > > types > > > > > of > > > > > > > > entity configs.. it's definitely more future proof. Thoughts? > > > > > > > > > > > > > > > > Aditya > > > > > > > > > > > > > > > > ________________________________________ > > > > > > > > From: Todd Palino [tpal...@gmail.com] > > > > > > > > Sent: Monday, May 18, 2015 12:39 PM > > > > > > > > To: dev@kafka.apache.org > > > > > > > > Subject: Re: [VOTE] KIP-21 Dynamic Configuration > > > > > > > > > > > > > > > > Agree with Jun here on the JSON format. I think your > intention > > > was > > > > > > likely > > > > > > > > to have actual JSON here and it was just a typo in the wiki? > > > > > > > > > > > > > > > > -Todd > > > > > > > > > > > > > > > > On Mon, May 18, 2015 at 12:07 PM, Jun Rao <j...@confluent.io> > > > > wrote: > > > > > > > > > > > > > > > > > Aditya, > > > > > > > > > > > > > > > > > > Another thing to consider. In KIP-4, we are adding a new > RPC > > > > > request > > > > > > to > > > > > > > > > change and retrieve topic configs. Do we want to add a > > similar > > > > RPC > > > > > > > > request > > > > > > > > > to change configs per client id? If so, do we want to > > > introduce a > > > > > > > > separate > > > > > > > > > new request or have a combined new request for both topic > and > > > > > client > > > > > > id > > > > > > > > > level config changes? > > > > > > > > > > > > > > > > > > A minor point in the wiki, for the json format in ZK, we > > should > > > > > > change > > > > > > > > > {X1=Y1, > > > > > > > > > X2=Y2..} to a json map, right? > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > > > Jun > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mon, May 18, 2015 at 9:48 AM, Aditya Auradkar < > > > > > > > > > aaurad...@linkedin.com.invalid> wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-21+-+Dynamic+Configuration > > > > > > > > > > > > > > > > > > > > Aditya > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > -- Guozhang > > >
