Hi Jay, Could you elaborate a bit more regarding 2) above, what is the motivation behind this?
On Wed, Mar 18, 2015 at 2:46 PM, Jay Kreps <jay.kr...@gmail.com> wrote: > The other question to figure out is the hierarchy of overrides. Currently > some configs can be overridden at the topic level using a fairly ad hoc > mechanism. I think Aditya's quota proposal requires having defaults and > overrides at the client.id level. You could imagine similar requirements > at > the user level when we have user as a concept. You could also imagine > overrides at a broker level (currently you do this by just changing the > config on that broker only). > > With respect to dynamic update: Some configs (e.g. socket buffers) CAN'T be > dynamically updated, and others (e.g. cleaner buffer) could theoretically > be updated by the implementation would be quite complicated. > > I think do really get this right: > 1. We would need to be able to mark configs as updatable or not (shouldn't > be too hard after the conversion to the config def stuff) in a way that > this automatically flows to the documentation. Obviously if you had to > guess which config was updatable that would be a nightmare. > 2. We will want to go with a pattern where instead of injecting individual > values into the objects that need them from the config we pass some kind of > Config object which has a reference to the current immutable config > instance. To get a config you always have to dereference > config.current.myConfigValue > so that when current gets updated you get all the new configs atomically. > 3. We will need to support a flexible notion of override that generalizes > the LogConfig stuff we did at the topic level. > > This is all a fair amount of work to do well. > > -Jay > > On Wed, Mar 18, 2015 at 10:11 AM, Andrii Biletskyi < > andrii.bilets...@stealth.ly> wrote: > > > Aditya, > > > > Yes, you are right. We agreed to think how global config can > > be done w/o restarting the broker, consumers/producers. I haven't > > yet updated the KIP accordingly though. > > In short, the problem is that KafkaConfig is "spread out" over different > > components, like SocketServer, OffsetManager etc. This makes > > it hard to dynamically change config, at least if we follow dynamic > > LogConfig approach. So my opinion and vision is that we need to > > treat each config block (like quotas in your case) separately. Logically, > > different cases will be handled differently, some of the configs are > > not eligible for dynamic change at all. > > > > Thanks, > > Andrii Biletskyi > > > > On Wed, Mar 18, 2015 at 6:54 PM, Aditya Auradkar < > > aaurad...@linkedin.com.invalid> wrote: > > > > > Hi Andrii, > > > > > > Thanks for the writeup. > > > IMO, we should be able to support dynamically changing certain configs. > > > For example: the Quota proposal relies on such a mechanism for changing > > > quotas on the fly. > > > > > > Aditya > > > ________________________________________ > > > From: Andrii Biletskyi [andrii.bilets...@stealth.ly] > > > Sent: Tuesday, March 03, 2015 10:30 AM > > > To: dev@kafka.apache.org > > > Subject: Re: [DISCUSS] KIP-5 - Broker Configuration Management > > > > > > Hey Jun, > > > > > > Initially I was thinking about instead of 3-4 state that global config > > > changes > > > take effect only after broker restart. So it's just: > > > > > > 3-4. On each broker startup apply global config from ZK > > > > > > In other words, the comprehensive workflow is the following: > > > 1. Issue ChangeGlobalConfigRequest > > > 2. Controller validates / stores config in ZK > > > (out of scope of this KIP) 3. Do rolling restart for brokers one by one > > to > > > pick up > > > config changes > > > > > > My understanding is that we won't be able to handle config change > > > dynamically > > > as we do for Log config. The main reason, from my point of view, is > that > > > broker config operates such settings as num.io.threads updating which > > would > > > mean > > > gracefully restart some of the broker's components (in this case > > > SocketServer) which is, > > > in turn, might be very tricky. > > > > > > Thanks, > > > Andrii Biletskyi > > > > > > On Tue, Mar 3, 2015 at 7:43 PM, Jun Rao <j...@confluent.io> wrote: > > > > > > > It seems the proposed workflow is the following. > > > > > > > > 1. Client issues a global config update request to the broker. > > > > 2. Broker writes the new config to ZK. > > > > 3. Controller picks up the changes from ZK. > > > > 4. Controller propagates the config changes to all brokers. > > > > > > > > Do we need to add a new request/response to propagate the config > > changes? > > > > > > > > Also, this logic is a bit different from how per topic config changes > > > > works: each broker reads from ZK directly. It would be good to make > > them > > > > consistent. > > > > > > > > Thanks, > > > > > > > > Jun > > > > > > > > > > > > On Wed, Jan 21, 2015 at 10:34 PM, Joe Stein <joe.st...@stealth.ly> > > > wrote: > > > > > > > > > Created a KIP > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-5+-+Broker+Configuration+Management > > > > > > > > > > JIRA https://issues.apache.org/jira/browse/KAFKA-1786 > > > > > > > > > > /******************************************* > > > > > Joe Stein > > > > > Founder, Principal Consultant > > > > > Big Data Open Source Security LLC > > > > > http://www.stealth.ly > > > > > Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop > > > > > > > ********************************************/ > > > > > > > > > > > > > > > -- -- Guozhang
