Awesome. +1 On Sunday, May 31, 2015, Aditya Auradkar <aaurad...@linkedin.com.invalid> wrote:
> 2. There was a typo in my previous email. I meant to say that we should > use snake case because it's more consistent. I couldn't find any examples > of camel case but did find some snake case (jmx_port). Other than that, > most other entries are single word keys. > > 3. The purge frequency is short (15 minutes). So it should be safe to > ignore older notifications. Add a story here: > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-21+-+Dynamic+Configuration#KIP-21-DynamicConfiguration-Migrationplanfornotifications > > Thanks, > Aditya > ________________________________________ > From: Jay Kreps [jay.kr...@gmail.com <javascript:;>] > Sent: Saturday, May 30, 2015 9:51 AM > To: dev@kafka.apache.org <javascript:;> > Subject: Re: [VOTE] KIP-21 Dynamic Configuration > > 1. Great. > 2. I don't have a preference as to the casing, but I really appreciate > consistency. Is everything using underscores today? If so let's stick with > that. If we are already inconsistent then I guess it's too late and we can > do whatever. Let me know and I'll update the coding standard. > 3. Not sure what the default purge frequency is. I don't think we need to > work the details of this out in the KIP, but we need a story for the > upgrade path so people don't get bitten. > > -Jay > > On Thu, May 28, 2015 at 11:22 PM, Aditya Auradkar < > aaurad...@linkedin.com.invalid> wrote: > > > Yeah, the same cleaning mechanism will be carried over. > > > > > 1. Are we introducing a new Java API for the config change protocol and > > if > > > so where will that appear? Is that going to be part of the java api in > > the > > > admin api kip? Let's document that. > > Yeah, we need to introduce a new Java API for the config change protocol. > > It should be a part of the AdminClient API. I'll alter KIP-4 to reflect > > that since the API is being introduced there. > > > > > 2. The proposed JSON format uses camel case for field names, is that > what > > > we've used for other JSON in zookeeper? > > I think camel case is more appropriate for the JSON format. For example, > > under the "brokers" znode, I found "jmx_port". > > > > > 3. This changes the format of the notifications, right? How will we > > > grandfather in the old format? Clusters will have existing change > > > notifications in the old format so I think the new code will need to be > > > able to read those? > > Interesting, I figured the existing notifications were purged by a > cleaner > > thread frequently. In that case, we wouldn't need to grandfather any > > notifications since we would only need to not make any config changes > for X > > minutes for there to be no changes in ZK. But the old notifications are > > actually removed when a new notification is received or the broker is > > bounced. So we do need to handle notifications in the old format. Should > we > > simply ignore older changes since they are only valid for a short period > of > > time? > > > > Thanks, > > Aditya > > ________________________________________ > > From: Jay Kreps [jay.kr...@gmail.com <javascript:;>] > > Sent: Thursday, May 28, 2015 5:25 PM > > To: dev@kafka.apache.org <javascript:;> > > Subject: Re: [VOTE] KIP-21 Dynamic Configuration > > > > That is handled now so I am assuming the same mechanism carries over? > > > > -Jay > > > > On Thu, May 28, 2015 at 5:12 PM, Guozhang Wang <wangg...@gmail.com > <javascript:;>> wrote: > > > > > For the sequential config/changes/config_change_XX znode, do we have > any > > > manners to do cleaning in order to avoid the change-log from growing > > > indefinitely? > > > > > > Guozhang > > > > > > On Thu, May 28, 2015 at 5:02 PM, Jay Kreps <jay.kr...@gmail.com > <javascript:;>> wrote: > > > > > > > I still have a couple of questions: > > > > 1. Are we introducing a new Java API for the config change protocol > and > > > if > > > > so where will that appear? Is that going to be part of the java api > in > > > the > > > > admin api kip? Let's document that. > > > > 2. The proposed JSON format uses camel case for field names, is that > > what > > > > we've used for other JSON in zookeeper? > > > > 3. This changes the format of the notifications, right? How will we > > > > grandfather in the old format? Clusters will have existing change > > > > notifications in the old format so I think the new code will need to > be > > > > able to read those? > > > > > > > > -Jay > > > > > > > > On Thu, May 28, 2015 at 11:41 AM, Aditya Auradkar < > > > > aaurad...@linkedin.com.invalid> wrote: > > > > > > > > > bump > > > > > > > > > > ________________________________________ > > > > > From: Aditya Auradkar > > > > > Sent: Tuesday, May 26, 2015 1:16 PM > > > > > To: dev@kafka.apache.org <javascript:;> > > > > > Subject: RE: [VOTE] KIP-21 Dynamic Configuration > > > > > > > > > > Hey everyone, > > > > > > > > > > Completed the changes to KIP-4. After today's hangout, there > doesn't > > > > > appear to be anything remaining to discuss on this KIP. > > > > > Please vote so we can formally close this. > > > > > > > > > > Thanks, > > > > > Aditya > > > > > > > > > > ________________________________________ > > > > > From: Aditya Auradkar > > > > > Sent: Thursday, May 21, 2015 11:26 AM > > > > > To: dev@kafka.apache.org <javascript:;> > > > > > Subject: RE: [VOTE] KIP-21 Dynamic Configuration > > > > > > > > > > I think we should remove the config part in TopicMetadataResponse. > > It's > > > > > probably cleaner if Alter and Describe are the only way to view and > > > > modify > > > > > configs but I don't feel very strongly about it. > > > > > > > > > > Re-summarizing the proposed changes to KIP-4: > > > > > - Change AlterTopic to not allow setting configs. Config changes > will > > > > flow > > > > > through AlterConfig. CreateTopic will still allow setting configs > as > > it > > > > is > > > > > nice to be able to specify configs while creating the topic. > > > > > - TopicMetadataResponse shoudn't return config for the topic. > > > > > DescribeConfig is the way to go. > > > > > - Change "InvalidTopicConfiguration" error code to > > > "InvalidEntityConfig" > > > > > as proposed in KIP-21. > > > > > > > > > > Aditya > > > > > > > > > > ________________________________________ > > > > > From: Jun Rao [j...@confluent.io <javascript:;>] > > > > > Sent: Thursday, May 21, 2015 10:50 AM > > > > > To: dev@kafka.apache.org <javascript:;> > > > > > Subject: Re: [VOTE] KIP-21 Dynamic Configuration > > > > > > > > > > What about TopicMetadataResponse in KIP-4? Do we remove the config > > part > > > > in > > > > > it? > > > > > > > > > > Thanks, > > > > > > > > > > Jun > > > > > > > > > > On Thu, May 21, 2015 at 10:25 AM, Aditya Auradkar < > > > > > aaurad...@linkedin.com.invalid> wrote: > > > > > > > > > > > Hey Jun, > > > > > > > > > > > > I've added a section on error codes on the KIP-21 wiki. > > > > > > > > > > > > Here are the proposed changes to KIP-4. I'll update the wiki > > shortly. > > > > > > - Change AlterTopic to not allow setting configs. Config changes > > will > > > > > flow > > > > > > through AlterConfig. CreateTopic will still allow setting configs > > as > > > it > > > > > is > > > > > > nice to be able to specify configs while creating the topic. > > > > > > - Change "InvalidTopicConfiguration" error code to > > > > "InvalidEntityConfig" > > > > > > as proposed in KIP-21. > > > > > > > > > > > > > > > > > > Thanks, > > > > > > Aditya > > > > > > > > > > > > ________________________________________ > > > > > > From: Jun Rao [j...@confluent.io <javascript:;>] > > > > > > Sent: Thursday, May 21, 2015 8:41 AM > > > > > > To: dev@kafka.apache.org <javascript:;> > > > > > > Subject: Re: [VOTE] KIP-21 Dynamic Configuration > > > > > > > > > > > > Aditya, > > > > > > > > > > > > For completeness, could you list the set of error codes in the > > wiki? > > > > > Also, > > > > > > could you summarize the changes that are needed for the requests > > > listed > > > > > in > > > > > > KIP-4 and update the wiki accordingly? > > > > > > > > > > > > Thanks, > > > > > > > > > > > > Jun > > > > > > > > > > > > On Tue, May 19, 2015 at 10:33 PM, Aditya Auradkar < > > > > > > aaurad...@linkedin.com.invalid> wrote: > > > > > > > > > > > > > Thanks Andrii. I'll make the changes. > > > > > > > > > > > > > > I've also updated KIP-21 to include the new config requests. > > Take a > > > > > look > > > > > > > and vote. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-21+-+Dynamic+Configuration > > > > > > > > > > > > > > Aditya > > > > > > > ________________________________________ > > > > > > > From: Andrii Biletskyi [andrii.bilets...@stealth.ly > <javascript:;>] > > > > > > > Sent: Tuesday, May 19, 2015 2:26 PM > > > > > > > To: dev@kafka.apache.org <javascript:;> > > > > > > > Subject: Re: [VOTE] KIP-21 Dynamic Configuration > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > Sorry I wasn't able to participate. I don't have objections > about > > > > > > removing > > > > > > > config changes from AlterTopic (as I understand both > AddedConfig > > > and > > > > > > > DeletedConfig) - you are welcome to update the KIP page. > > > > > > > > > > > > > > Thanks, > > > > > > > Andrii Biletskyi > > > > > > > > > > > > > > On Tue, May 19, 2015 at 11:40 PM, Aditya Auradkar < > > > > > > > aaurad...@linkedin.com.invalid> wrote: > > > > > > > > > > > > > > > Updating the discussion with the latest comments. > > > > > > > > > > > > > > > > 1. We discussed adding 2 new API's (AlterConfig and > > > > DescribeConfig). > > > > > > I'll > > > > > > > > update KIP-21 with details on these. > > > > > > > > 2. Discussed during the KIP hangout. We are in agreement. > > > > > > > > > > > > > > > > (1) has a dependency on KIP-4 being completed. Rest of the > work > > > in > > > > > the > > > > > > > KIP > > > > > > > > can be implemented independently. Any concerns if we tackle > it > > as > > > > two > > > > > > > > separate work items implementation wise? > > > > > > > > > > > > > > > > We also discussed changing the AlterTopic command in KIP-4 to > > not > > > > > > include > > > > > > > > config changes. Instead, all config changes will pass through > > the > > > > > newly > > > > > > > > proposed AlterConfig. If no-one objects, I can make some > > changes > > > to > > > > > > KIP-4 > > > > > > > > to reflect this. > > > > > > > > > > > > > > > > Aditya > > > > > > > > > > > > > > > > ________________________________________ > > > > > > > > From: Jay Kreps [jay.kr...@gmail.com <javascript:;>] > > > > > > > > Sent: Tuesday, May 19, 2015 10:51 AM > > > > > > > > To: dev@kafka.apache.org <javascript:;> > > > > > > > > Subject: Re: [VOTE] KIP-21 Dynamic Configuration > > > > > > > > > > > > > > > > Hey Aditya, > > > > > > > > > > > > > > > > Two comments: > > > > > > > > > > > > > > > > 1. Yeah we need to reconcile this with the APIs in KIP-4. I > > think > > > > it > > > > > > does > > > > > > > > make sense to allow setting config during topic creation. I > > agree > > > > > with > > > > > > > your > > > > > > > > summary that having alter topic and alter config may be > > > confusing, > > > > > but > > > > > > > > there are also some non-config changes such as replication > > factor > > > > and > > > > > > > > partition count that alter topic can carry out. What is the > > final > > > > > state > > > > > > > you > > > > > > > > are proposing? > > > > > > > > > > > > > > > > 2. This is implementation related so probably can be removed > > from > > > > the > > > > > > KIP > > > > > > > > entirely, but you seem to be proposing a separate config > > manager > > > > for > > > > > > each > > > > > > > > config override type. Should we just generalize > > > TopicConfigManager > > > > to > > > > > > be > > > > > > > > ConfigOverrideManager and have it handle all the override > types > > > we > > > > > will > > > > > > > > have? I think I may just be unclear on what you are > > proposing... > > > > > > > > > > > > > > > > -Jay > > > > > > > > > > > > > > > > On Mon, May 18, 2015 at 1:34 PM, Aditya Auradkar < > > > > > > > > aaurad...@linkedin.com.invalid> wrote: > > > > > > > > > > > > > > > > > Yeah, that was just a typo. I've fixed it. Thanks for > calling > > > it > > > > > out. > > > > > > > > > > > > > > > > > > In KIP-4, I believe we have 3 types of requests: > CreateTopic, > > > > > > > AlterTopic > > > > > > > > > and DeleteTopic. The topic configs are a sub-type of the > > Create > > > > and > > > > > > > Alter > > > > > > > > > commands. I think it would be nice to simply have a > > AlterConfig > > > > > > command > > > > > > > > > that can alter any type of config rather than having a > > specific > > > > > > > > > ClientConfig. > > > > > > > > > > > > > > > > > > AlterConfig => [ConfigType [AddedConfigEntry] > > [DeletedConfig]] > > > > > > > > > ConfigType => string > > > > > > > > > AddedConfigEntry => ConfigKey ConfigValue > > > > > > > > > ConfigKey => string > > > > > > > > > ConfigValue => string > > > > > > > > > DeletedConfig => string > > > > > > > > > > > > > > > > > > The downside of this approach is that we will have 2 > separate > > > > ways > > > > > of > > > > > > > > > changing topic configs (AlterTopic and AlterConfig). While > a > > > > > general > > > > > > > > > AlterConfig only makes sense if we plan to have more than > two > > > > types > > > > > > of > > > > > > > > > entity configs.. it's definitely more future proof. > Thoughts? > > > > > > > > > > > > > > > > > > Aditya > > > > > > > > > > > > > > > > > > ________________________________________ > > > > > > > > > From: Todd Palino [tpal...@gmail.com <javascript:;>] > > > > > > > > > Sent: Monday, May 18, 2015 12:39 PM > > > > > > > > > To: dev@kafka.apache.org <javascript:;> > > > > > > > > > Subject: Re: [VOTE] KIP-21 Dynamic Configuration > > > > > > > > > > > > > > > > > > Agree with Jun here on the JSON format. I think your > > intention > > > > was > > > > > > > likely > > > > > > > > > to have actual JSON here and it was just a typo in the > wiki? > > > > > > > > > > > > > > > > > > -Todd > > > > > > > > > > > > > > > > > > On Mon, May 18, 2015 at 12:07 PM, Jun Rao < > j...@confluent.io <javascript:;>> > > > > > wrote: > > > > > > > > > > > > > > > > > > > Aditya, > > > > > > > > > > > > > > > > > > > > Another thing to consider. In KIP-4, we are adding a new > > RPC > > > > > > request > > > > > > > to > > > > > > > > > > change and retrieve topic configs. Do we want to add a > > > similar > > > > > RPC > > > > > > > > > request > > > > > > > > > > to change configs per client id? If so, do we want to > > > > introduce a > > > > > > > > > separate > > > > > > > > > > new request or have a combined new request for both topic > > and > > > > > > client > > > > > > > id > > > > > > > > > > level config changes? > > > > > > > > > > > > > > > > > > > > A minor point in the wiki, for the json format in ZK, we > > > should > > > > > > > change > > > > > > > > > > {X1=Y1, > > > > > > > > > > X2=Y2..} to a json map, right? > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > > > > > Jun > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mon, May 18, 2015 at 9:48 AM, Aditya Auradkar < > > > > > > > > > > aaurad...@linkedin.com.invalid> wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-21+-+Dynamic+Configuration > > > > > > > > > > > > > > > > > > > > > > Aditya > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > -- Guozhang > > > > > >
