Re: [DISCUSS] KIP-5 - Broker Configuration Management

2015-03-20 Thread Guozhang Wang
writeup. > > > IMO, we should be able to support dynamically changing certain configs. > > > For example: the Quota proposal relies on such a mechanism for changing > > > quotas on the fly. > > > > > > Aditya > > > __

Re: [DISCUSS] KIP-5 - Broker Configuration Management

2015-03-18 Thread Jay Kreps
configs. > > For example: the Quota proposal relies on such a mechanism for changing > > quotas on the fly. > > > > Aditya > > ____ > > From: Andrii Biletskyi [andrii.bilets...@stealth.ly] > > Sent: Tuesday, March 03, 2015 10:30 AM > > To: dev@kafka.ap

Re: [DISCUSS] KIP-5 - Broker Configuration Management

2015-03-18 Thread Andrii Biletskyi
r example: the Quota proposal relies on such a mechanism for changing > quotas on the fly. > > Aditya > > From: Andrii Biletskyi [andrii.bilets...@stealth.ly] > Sent: Tuesday, March 03, 2015 10:30 AM > To: dev@kafka.apache.org > Su

RE: [DISCUSS] KIP-5 - Broker Configuration Management

2015-03-18 Thread Aditya Auradkar
...@stealth.ly] Sent: Tuesday, March 03, 2015 10:30 AM To: dev@kafka.apache.org Subject: Re: [DISCUSS] KIP-5 - Broker Configuration Management Hey Jun, Initially I was thinking about instead of 3-4 state that global config changes take effect only after broker restart. So it's just: 3-4. On each b

Re: [DISCUSS] KIP-5 - Broker Configuration Management

2015-03-03 Thread Andrii Biletskyi
Hey Jun, Initially I was thinking about instead of 3-4 state that global config changes take effect only after broker restart. So it's just: 3-4. On each broker startup apply global config from ZK In other words, the comprehensive workflow is the following: 1. Issue ChangeGlobalConfigRequest 2.

Re: [DISCUSS] KIP-5 - Broker Configuration Management

2015-03-03 Thread Jun Rao
It seems the proposed workflow is the following. 1. Client issues a global config update request to the broker. 2. Broker writes the new config to ZK. 3. Controller picks up the changes from ZK. 4. Controller propagates the config changes to all brokers. Do we need to add a new request/response t

Re: [DISCUSS] KIP-5 - Broker Configuration Management

2015-02-22 Thread Neha Narkhede
Andrii, Thanks for updating the KIP. Few comments - Would you mind updating the Compatibility, Migration Plan section as well? We should discuss the new config store options - I'm not so sure that zookeeper is such a slam dunk choice for storing the configs. 1. Firstly, we have seen quite a

Re: [DISCUSS] KIP-5 - Broker Configuration Management

2015-02-16 Thread Andrii Biletskyi
Hi all, I've added some details to Proposed Changes paragraph. https://cwiki.apache.org/confluence/display/KAFKA/KIP-5+-+Broker+Configuration+Management Please let me know if something else require description / details. Thanks, Andrii On Sat, Jan 24, 2015 at 1:39 AM, Jay Kreps wrote: > Cool.

Re: [DISCUSS] KIP-5 - Broker Configuration Management

2015-01-23 Thread Jay Kreps
Cool. Yeah sorry to nag about these KIPs, and I hope it doesn't come across the wrong way. But the hope I really have for these is that they are complete enough that even highly involved users can see and understand the change, motivation, etc. I think that will do a ton to help extend the communit

Re: [DISCUSS] KIP-5 - Broker Configuration Management

2015-01-22 Thread Joe Stein
There is still some to-dos to be done in https://reviews.apache.org/r/29513/diff/ to use changing to ConfigDef https://reviews.apache.org/r/30126/diff/ once that is in. We can get more written up on it, will do. On Fri, Jan 23, 2015 at 12:05 AM, Jay Kreps wrote: > Hey Joe, > > Can you fill in t

Re: [DISCUSS] KIP-5 - Broker Configuration Management

2015-01-22 Thread Jay Kreps
Hey Joe, Can you fill in this KIP? The purpose of these KIPs is to give a full overview of the feature, how it will work, be implemented, the considerations involved, etc. There is only like one sentence on this which isn't enough for anyone to know what you are thinking. Moving off of configs to

Re: [DISCUSS] KIP-5 - Broker Configuration Management

2015-01-22 Thread Gwen Shapira
Thanks for clarifying the upgrade path. +1 for this proposal, it will greatly reduce errors resulting from configuration mismatches. On Thu, Jan 22, 2015 at 8:40 AM, Joe Stein wrote: > Deployment proposal: > >1. If global configuration isn't being used, do what you do today (there >shou

Re: [DISCUSS] KIP-5 - Broker Configuration Management

2015-01-22 Thread Joe Stein
Deployment proposal: 1. If global configuration isn't being used, do what you do today (there should be configuration brokers.admin.global.configuration=true [default=false]. This works for the upgrade. Once all brokers are upgraded proceed. 2. if the global config is used then only

Re: [DISCUSS] KIP-5 - Broker Configuration Management

2015-01-22 Thread Harsha
Hi Joe, How does initial deployments will look in this environment. I guess users will deploy the kafka cluster and it will be running with defaults and they can use a command line tool to update the configs?. I think you need to update the JIRA and mailing list thread links on the

[DISCUSS] KIP-5 - Broker Configuration Management

2015-01-21 Thread Joe Stein
Created a KIP https://cwiki.apache.org/confluence/display/KAFKA/KIP-5+-+Broker+Configuration+Management JIRA https://issues.apache.org/jira/browse/KAFKA-1786 /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.steal