The other question to figure out is the hierarchy of overrides. Currently
some configs can be overridden at the topic level using a fairly ad hoc
mechanism. I think Aditya's quota proposal requires having defaults and
overrides at the client.id level. You could imagine similar requirements at
the user level when we have user as a concept. You could also imagine
overrides at a broker level (currently you do this by just changing the
config on that broker only).
With respect to dynamic update: Some configs (e.g. socket buffers) CAN'T be
dynamically updated, and others (e.g. cleaner buffer) could theoretically
be updated by the implementation would be quite complicated.
I think do really get this right:
1. We would need to be able to mark configs as updatable or not (shouldn't
be too hard after the conversion to the config def stuff) in a way that
this automatically flows to the documentation. Obviously if you had to
guess which config was updatable that would be a nightmare.
2. We will want to go with a pattern where instead of injecting individual
values into the objects that need them from the config we pass some kind of
Config object which has a reference to the current immutable config
instance. To get a config you always have to dereference
config.current.myConfigValue
so that when current gets updated you get all the new configs atomically.
3. We will need to support a flexible notion of override that generalizes
the LogConfig stuff we did at the topic level.
This is all a fair amount of work to do well.
-Jay
On Wed, Mar 18, 2015 at 10:11 AM, Andrii Biletskyi <
andrii.bilets...@stealth.ly> wrote:
> Aditya,
>
> Yes, you are right. We agreed to think how global config can
> be done w/o restarting the broker, consumers/producers. I haven't
> yet updated the KIP accordingly though.
> In short, the problem is that KafkaConfig is "spread out" over different
> components, like SocketServer, OffsetManager etc. This makes
> it hard to dynamically change config, at least if we follow dynamic
> LogConfig approach. So my opinion and vision is that we need to
> treat each config block (like quotas in your case) separately. Logically,
> different cases will be handled differently, some of the configs are
> not eligible for dynamic change at all.
>
> Thanks,
> Andrii Biletskyi
>
> On Wed, Mar 18, 2015 at 6:54 PM, Aditya Auradkar <
> aaurad...@linkedin.com.invalid> wrote:
>
> > Hi Andrii,
> >
> > Thanks for the writeup.
> > IMO, we should be able to support dynamically changing certain configs.
> > For example: the Quota proposal relies on such a mechanism for changing
> > quotas on the fly.
> >
> > Aditya
> > ________________________________________
> > From: Andrii Biletskyi [andrii.bilets...@stealth.ly]
> > Sent: Tuesday, March 03, 2015 10:30 AM
> > To: dev@kafka.apache.org
> > Subject: Re: [DISCUSS] KIP-5 - Broker Configuration Management
> >
> > Hey Jun,
> >
> > Initially I was thinking about instead of 3-4 state that global config
> > changes
> > take effect only after broker restart. So it's just:
> >
> > 3-4. On each broker startup apply global config from ZK
> >
> > In other words, the comprehensive workflow is the following:
> > 1. Issue ChangeGlobalConfigRequest
> > 2. Controller validates / stores config in ZK
> > (out of scope of this KIP) 3. Do rolling restart for brokers one by one
> to
> > pick up
> > config changes
> >
> > My understanding is that we won't be able to handle config change
> > dynamically
> > as we do for Log config. The main reason, from my point of view, is that
> > broker config operates such settings as num.io.threads updating which
> would
> > mean
> > gracefully restart some of the broker's components (in this case
> > SocketServer) which is,
> > in turn, might be very tricky.
> >
> > Thanks,
> > Andrii Biletskyi
> >
> > On Tue, Mar 3, 2015 at 7:43 PM, Jun Rao <j...@confluent.io> wrote:
> >
> > > It seems the proposed workflow is the following.
> > >
> > > 1. Client issues a global config update request to the broker.
> > > 2. Broker writes the new config to ZK.
> > > 3. Controller picks up the changes from ZK.
> > > 4. Controller propagates the config changes to all brokers.
> > >
> > > Do we need to add a new request/response to propagate the config
> changes?
> > >
> > > Also, this logic is a bit different from how per topic config changes
> > > works: each broker reads from ZK directly. It would be good to make
> them
> > > consistent.
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > >
> > > On Wed, Jan 21, 2015 at 10:34 PM, Joe Stein <joe.st...@stealth.ly>
> > wrote:
> > >
> > > > Created a KIP
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-5+-+Broker+Configuration+Management
> > > >
> > > > JIRA https://issues.apache.org/jira/browse/KAFKA-1786
> > > >
> > > > /*******************************************
> > > > Joe Stein
> > > > Founder, Principal Consultant
> > > > Big Data Open Source Security LLC
> > > > http://www.stealth.ly
> > > > Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
> > > > ********************************************/
> > > >
> > >
> >
>
