Minor edit: I meant that we should expect change notifications in the old format made earlier, but should perhaps ignore them. After the upgrade is done, older versions of AdminTools can no longer be used to make config changes.
Aditya ________________________________________ From: Aditya Auradkar Sent: Thursday, May 28, 2015 11:22 PM To: dev@kafka.apache.org Subject: RE: [VOTE] KIP-21 Dynamic Configuration Yeah, the same cleaning mechanism will be carried over. > 1. Are we introducing a new Java API for the config change protocol and if > so where will that appear? Is that going to be part of the java api in the > admin api kip? Let's document that. Yeah, we need to introduce a new Java API for the config change protocol. It should be a part of the AdminClient API. I'll alter KIP-4 to reflect that since the API is being introduced there. > 2. The proposed JSON format uses camel case for field names, is that what > we've used for other JSON in zookeeper? I think camel case is more appropriate for the JSON format. For example, under the "brokers" znode, I found "jmx_port". > 3. This changes the format of the notifications, right? How will we > grandfather in the old format? Clusters will have existing change > notifications in the old format so I think the new code will need to be > able to read those? Interesting, I figured the existing notifications were purged by a cleaner thread frequently. In that case, we wouldn't need to grandfather any notifications since we would only need to not make any config changes for X minutes for there to be no changes in ZK. But the old notifications are actually removed when a new notification is received or the broker is bounced. So we do need to handle notifications in the old format. Should we simply ignore older changes since they are only valid for a short period of time? Thanks, Aditya ________________________________________ From: Jay Kreps [jay.kr...@gmail.com] Sent: Thursday, May 28, 2015 5:25 PM To: dev@kafka.apache.org Subject: Re: [VOTE] KIP-21 Dynamic Configuration That is handled now so I am assuming the same mechanism carries over? -Jay On Thu, May 28, 2015 at 5:12 PM, Guozhang Wang <wangg...@gmail.com> wrote: > For the sequential config/changes/config_change_XX znode, do we have any > manners to do cleaning in order to avoid the change-log from growing > indefinitely? > > Guozhang > > On Thu, May 28, 2015 at 5:02 PM, Jay Kreps <jay.kr...@gmail.com> wrote: > > > I still have a couple of questions: > > 1. Are we introducing a new Java API for the config change protocol and > if > > so where will that appear? Is that going to be part of the java api in > the > > admin api kip? Let's document that. > > 2. The proposed JSON format uses camel case for field names, is that what > > we've used for other JSON in zookeeper? > > 3. This changes the format of the notifications, right? How will we > > grandfather in the old format? Clusters will have existing change > > notifications in the old format so I think the new code will need to be > > able to read those? > > > > -Jay > > > > On Thu, May 28, 2015 at 11:41 AM, Aditya Auradkar < > > aaurad...@linkedin.com.invalid> wrote: > > > > > bump > > > > > > ________________________________________ > > > From: Aditya Auradkar > > > Sent: Tuesday, May 26, 2015 1:16 PM > > > To: dev@kafka.apache.org > > > Subject: RE: [VOTE] KIP-21 Dynamic Configuration > > > > > > Hey everyone, > > > > > > Completed the changes to KIP-4. After today's hangout, there doesn't > > > appear to be anything remaining to discuss on this KIP. > > > Please vote so we can formally close this. > > > > > > Thanks, > > > Aditya > > > > > > ________________________________________ > > > From: Aditya Auradkar > > > Sent: Thursday, May 21, 2015 11:26 AM > > > To: dev@kafka.apache.org > > > Subject: RE: [VOTE] KIP-21 Dynamic Configuration > > > > > > I think we should remove the config part in TopicMetadataResponse. It's > > > probably cleaner if Alter and Describe are the only way to view and > > modify > > > configs but I don't feel very strongly about it. > > > > > > Re-summarizing the proposed changes to KIP-4: > > > - Change AlterTopic to not allow setting configs. Config changes will > > flow > > > through AlterConfig. CreateTopic will still allow setting configs as it > > is > > > nice to be able to specify configs while creating the topic. > > > - TopicMetadataResponse shoudn't return config for the topic. > > > DescribeConfig is the way to go. > > > - Change "InvalidTopicConfiguration" error code to > "InvalidEntityConfig" > > > as proposed in KIP-21. > > > > > > Aditya > > > > > > ________________________________________ > > > From: Jun Rao [j...@confluent.io] > > > Sent: Thursday, May 21, 2015 10:50 AM > > > To: dev@kafka.apache.org > > > Subject: Re: [VOTE] KIP-21 Dynamic Configuration > > > > > > What about TopicMetadataResponse in KIP-4? Do we remove the config part > > in > > > it? > > > > > > Thanks, > > > > > > Jun > > > > > > On Thu, May 21, 2015 at 10:25 AM, Aditya Auradkar < > > > aaurad...@linkedin.com.invalid> wrote: > > > > > > > Hey Jun, > > > > > > > > I've added a section on error codes on the KIP-21 wiki. > > > > > > > > Here are the proposed changes to KIP-4. I'll update the wiki shortly. > > > > - Change AlterTopic to not allow setting configs. Config changes will > > > flow > > > > through AlterConfig. CreateTopic will still allow setting configs as > it > > > is > > > > nice to be able to specify configs while creating the topic. > > > > - Change "InvalidTopicConfiguration" error code to > > "InvalidEntityConfig" > > > > as proposed in KIP-21. > > > > > > > > > > > > Thanks, > > > > Aditya > > > > > > > > ________________________________________ > > > > From: Jun Rao [j...@confluent.io] > > > > Sent: Thursday, May 21, 2015 8:41 AM > > > > To: dev@kafka.apache.org > > > > Subject: Re: [VOTE] KIP-21 Dynamic Configuration > > > > > > > > Aditya, > > > > > > > > For completeness, could you list the set of error codes in the wiki? > > > Also, > > > > could you summarize the changes that are needed for the requests > listed > > > in > > > > KIP-4 and update the wiki accordingly? > > > > > > > > Thanks, > > > > > > > > Jun > > > > > > > > On Tue, May 19, 2015 at 10:33 PM, Aditya Auradkar < > > > > aaurad...@linkedin.com.invalid> wrote: > > > > > > > > > Thanks Andrii. I'll make the changes. > > > > > > > > > > I've also updated KIP-21 to include the new config requests. Take a > > > look > > > > > and vote. > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-21+-+Dynamic+Configuration > > > > > > > > > > Aditya > > > > > ________________________________________ > > > > > From: Andrii Biletskyi [andrii.bilets...@stealth.ly] > > > > > Sent: Tuesday, May 19, 2015 2:26 PM > > > > > To: dev@kafka.apache.org > > > > > Subject: Re: [VOTE] KIP-21 Dynamic Configuration > > > > > > > > > > Hi, > > > > > > > > > > Sorry I wasn't able to participate. I don't have objections about > > > > removing > > > > > config changes from AlterTopic (as I understand both AddedConfig > and > > > > > DeletedConfig) - you are welcome to update the KIP page. > > > > > > > > > > Thanks, > > > > > Andrii Biletskyi > > > > > > > > > > On Tue, May 19, 2015 at 11:40 PM, Aditya Auradkar < > > > > > aaurad...@linkedin.com.invalid> wrote: > > > > > > > > > > > Updating the discussion with the latest comments. > > > > > > > > > > > > 1. We discussed adding 2 new API's (AlterConfig and > > DescribeConfig). > > > > I'll > > > > > > update KIP-21 with details on these. > > > > > > 2. Discussed during the KIP hangout. We are in agreement. > > > > > > > > > > > > (1) has a dependency on KIP-4 being completed. Rest of the work > in > > > the > > > > > KIP > > > > > > can be implemented independently. Any concerns if we tackle it as > > two > > > > > > separate work items implementation wise? > > > > > > > > > > > > We also discussed changing the AlterTopic command in KIP-4 to not > > > > include > > > > > > config changes. Instead, all config changes will pass through the > > > newly > > > > > > proposed AlterConfig. If no-one objects, I can make some changes > to > > > > KIP-4 > > > > > > to reflect this. > > > > > > > > > > > > Aditya > > > > > > > > > > > > ________________________________________ > > > > > > From: Jay Kreps [jay.kr...@gmail.com] > > > > > > Sent: Tuesday, May 19, 2015 10:51 AM > > > > > > To: dev@kafka.apache.org > > > > > > Subject: Re: [VOTE] KIP-21 Dynamic Configuration > > > > > > > > > > > > Hey Aditya, > > > > > > > > > > > > Two comments: > > > > > > > > > > > > 1. Yeah we need to reconcile this with the APIs in KIP-4. I think > > it > > > > does > > > > > > make sense to allow setting config during topic creation. I agree > > > with > > > > > your > > > > > > summary that having alter topic and alter config may be > confusing, > > > but > > > > > > there are also some non-config changes such as replication factor > > and > > > > > > partition count that alter topic can carry out. What is the final > > > state > > > > > you > > > > > > are proposing? > > > > > > > > > > > > 2. This is implementation related so probably can be removed from > > the > > > > KIP > > > > > > entirely, but you seem to be proposing a separate config manager > > for > > > > each > > > > > > config override type. Should we just generalize > TopicConfigManager > > to > > > > be > > > > > > ConfigOverrideManager and have it handle all the override types > we > > > will > > > > > > have? I think I may just be unclear on what you are proposing... > > > > > > > > > > > > -Jay > > > > > > > > > > > > On Mon, May 18, 2015 at 1:34 PM, Aditya Auradkar < > > > > > > aaurad...@linkedin.com.invalid> wrote: > > > > > > > > > > > > > Yeah, that was just a typo. I've fixed it. Thanks for calling > it > > > out. > > > > > > > > > > > > > > In KIP-4, I believe we have 3 types of requests: CreateTopic, > > > > > AlterTopic > > > > > > > and DeleteTopic. The topic configs are a sub-type of the Create > > and > > > > > Alter > > > > > > > commands. I think it would be nice to simply have a AlterConfig > > > > command > > > > > > > that can alter any type of config rather than having a specific > > > > > > > ClientConfig. > > > > > > > > > > > > > > AlterConfig => [ConfigType [AddedConfigEntry] [DeletedConfig]] > > > > > > > ConfigType => string > > > > > > > AddedConfigEntry => ConfigKey ConfigValue > > > > > > > ConfigKey => string > > > > > > > ConfigValue => string > > > > > > > DeletedConfig => string > > > > > > > > > > > > > > The downside of this approach is that we will have 2 separate > > ways > > > of > > > > > > > changing topic configs (AlterTopic and AlterConfig). While a > > > general > > > > > > > AlterConfig only makes sense if we plan to have more than two > > types > > > > of > > > > > > > entity configs.. it's definitely more future proof. Thoughts? > > > > > > > > > > > > > > Aditya > > > > > > > > > > > > > > ________________________________________ > > > > > > > From: Todd Palino [tpal...@gmail.com] > > > > > > > Sent: Monday, May 18, 2015 12:39 PM > > > > > > > To: dev@kafka.apache.org > > > > > > > Subject: Re: [VOTE] KIP-21 Dynamic Configuration > > > > > > > > > > > > > > Agree with Jun here on the JSON format. I think your intention > > was > > > > > likely > > > > > > > to have actual JSON here and it was just a typo in the wiki? > > > > > > > > > > > > > > -Todd > > > > > > > > > > > > > > On Mon, May 18, 2015 at 12:07 PM, Jun Rao <j...@confluent.io> > > > wrote: > > > > > > > > > > > > > > > Aditya, > > > > > > > > > > > > > > > > Another thing to consider. In KIP-4, we are adding a new RPC > > > > request > > > > > to > > > > > > > > change and retrieve topic configs. Do we want to add a > similar > > > RPC > > > > > > > request > > > > > > > > to change configs per client id? If so, do we want to > > introduce a > > > > > > > separate > > > > > > > > new request or have a combined new request for both topic and > > > > client > > > > > id > > > > > > > > level config changes? > > > > > > > > > > > > > > > > A minor point in the wiki, for the json format in ZK, we > should > > > > > change > > > > > > > > {X1=Y1, > > > > > > > > X2=Y2..} to a json map, right? > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > Jun > > > > > > > > > > > > > > > > > > > > > > > > On Mon, May 18, 2015 at 9:48 AM, Aditya Auradkar < > > > > > > > > aaurad...@linkedin.com.invalid> wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-21+-+Dynamic+Configuration > > > > > > > > > > > > > > > > > > Aditya > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > -- Guozhang >
