oudera.com]
> Sent: Friday, May 15, 2015 12:55 PM
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-21 Configuration Management
>
> The wiki says:
> "There will be 3 paths within config
> /config/clients/
> /config/topics/
> /config/brokers/
> Didn't we d
Yes we did. I just overlooked that line.. cleaning it up now.
Aditya
From: Gwen Shapira [gshap...@cloudera.com]
Sent: Friday, May 15, 2015 12:55 PM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-21 Configuration Management
The wiki says:
"There
ail.com]
> Sent: Tuesday, May 12, 2015 1:09 PM
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-21 Configuration Management
>
> The lack of audit could be addressed to some degree by an internal
> __config_changes topic which can have very long retention. Also, per
> t
@kafka.apache.org
Subject: Re: [DISCUSS] KIP-21 Configuration Management
The lack of audit could be addressed to some degree by an internal
__config_changes topic which can have very long retention. Also, per
the hangout summary that Gwen sent out it appears that we decided
against supporting SIGHUP/dynamic
t; > > > > > > > > > >
> > > > > > > > > > > On Tue, May 5, 2015 at 4:14 PM, Gwen Shapira <
> > > > > > > gshap...@cloudera.com >
> > > > > >
Make sure that traditional deployment tools (Puppet,
> > Chef,
> > > > etc)
> > > > > > are
> > > > > > > > > still
> > > > > > > > > > > capable of managing Kafka configuration.
> > > > > > > > > > >
> &
this involves handling HUP signal in the main
> thread
> > > to
> > > > > > reload
> > > > > > > > > > configuration. Then packaging scripts can add something
> > nice
> > > like
> > > > > > > > > "service
> >
015 at 8:54 AM, Joel Koshy <
> > jjkosh...@gmail.com >
> > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Good discussion. Since we will be talking about this at
> > 11am, I
> > > > > &
nfig changes. This
> > > needs
> > > > > to
> > > > > > > > > be general enough to work for all configs that we envision
> may
> > > > need
> > > > > > to
> > > > &
.
Aditya
From: Joel Koshy [jjkosh...@gmail.com]
Sent: Monday, May 11, 2015 4:54 PM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-21 Configuration Management
So the general concern here is the dichotomy of configs (which we
already have - i.e., in the form of broker config files vs topic
)
> > > > > > > >
> > > > > > > > The current KIP is really focused on REQUIREMENT 1 and I think
> > > that
> > > > > is
> > > > > > > > reasonable as long as we don't come up with
t; > > > > >
> > > > > > > REQUIREMENT 2: Provide consistency of configs across brokers
> > > (modulo
> > > > > > > per-broker overrides) or at least be able to verify
> consistency.
> > > > What
> > > > > > > this effectively means is that config changes
> > > > > >
> > > > > > REQUIREMENT 3: Central config store. Needs to work with plain
> > > > > > file-based configs and other systems (e.g., puppet). Ideally,
> > should
> > > > > > not bring in other dependencies (e.g., a DB). Possible options:
> > > > > > -
puppet). Ideally,
> > should
> > > > > > not bring in other dependencies (e.g., a DB). Possible options:
> > > > > > - ZooKeeper
> > > > > > - Kafka topic
> > > > > > - other? E.g. making it pluggable?
> > > &
; > Joel
> > > > >
> > > > > On Tue, May 05, 2015 at 01:38:09AM +0000, Aditya Auradkar wrote:
> > > > > > Hey Neha,
> > > > > >
> > > > > > Thanks for the feedback.
> > > > > > 1. In my earlier
ng
> > all
> > > > it's configs to ZK (while respecting the overrides). Then ZK can be
> > used
> > > to
> > > > view all configs.
> > > > >
> > > > > 2. Need to think about this a bit more. Perhaps we can discuss this
> > > > during the hang
t; > > operations. In the case, it may be reasonable to assume that the ZK
> port
> > is
> > > available for communication from the machine these commands are run.
> > Having
> > > a ConfigChangeRequest (or similar) is nice to have but having a new API
> >
roker/controller API.
Aditya
From: Ashish Singh [asi...@cloudera.com]
Sent: Thursday, May 07, 2015 8:19 AM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-21 Configuration Management
Agreed :). However, the other concerns remain. Do you think just prov
> > REQUIREMENT 2: Provide consistency of configs across brokers
> > > (modulo
> > > > > > > per-broker overrides) or at least be able to verify
> consistency.
> > > > What
> > > > > > > this effectively means is that config changes
t; > > > > > not bring in other dependencies (e.g., a DB). Possible options:
> > > > > > - ZooKeeper
> > > > > > - Kafka topic
> > > > > > - other? E.g. making it pluggable?
> > > > > >
> > > > > > Any other r
t;> > > > > REQUIREMENT 3: Central config store. Needs to work with plain
>> > > > > file-based configs and other systems (e.g., puppet). Ideally,
>> should
>> > > > > not bring in other dependencies (e.g., a DB). Possible options:
>> > > > > - ZooKeeper
>> > >
> > Any other requirements?
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Joel
> > > > >
> > > > > On Tue, May 05, 2015 at 01:38:09AM +0000, Aditya Auradkar wrote:
> > > > > > Hey Neha,
> > > > >
; > > > 1. In my earlier exchange with Jay, I mentioned the broker writing
> > all
> > > > it's configs to ZK (while respecting the overrides). Then ZK can be
> > used
> > > to
> > > > view all configs.
> > > > >
> >
t; port
> > is
> > > available for communication from the machine these commands are run.
> > Having
> > > a ConfigChangeRequest (or similar) is nice to have but having a new API
> > and
> > > sending requests to controller also change how we do topic based
nfigChangeRequest (or similar) is nice to have but having a new API
> and
> > sending requests to controller also change how we do topic based
> > configuration currently. I was hoping to keep this KIP as minimal as
> > possible and provide a means to represent and modify client and b
I was hoping to keep this KIP as minimal as
> possible and provide a means to represent and modify client and broker
> based configs in a central place. Are there any concerns if we tackle these
> things in a later KIP?
> >
> > Thanks,
> > Aditya
> > _
ient and broker
> based configs in a central place. Are there any concerns if we tackle these
> things in a later KIP?
> >
> > Thanks,
> > Aditya
> >
> > From: Neha Narkhede [n...@confluent.io]
> > Sent: Sunday, May 03
;
> From: Neha Narkhede [n...@confluent.io]
> Sent: Sunday, May 03, 2015 9:48 AM
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-21 Configuration Management
>
> Thanks for starting this discussion, Aditya. Few questions/comments
>
n a later KIP?
Thanks,
Aditya
From: Neha Narkhede [n...@confluent.io]
Sent: Sunday, May 03, 2015 9:48 AM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-21 Configuration Management
Thanks for starting this discussion, Aditya. Few questions/comments
scope.
Thanks,
Aditya
From: Jay Kreps [jay.kr...@gmail.com]
Sent: Monday, May 04, 2015 2:00 PM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-21 Configuration Management
Hey Aditya,
1. I would argue for either staying with what we have or else movi
; > notification or a request type exposed by each broker?
> >
> > Thanks,
> > Aditya
> >
> >
> > From: Joe Stein [joe.st...@stealth.ly]
> > Sent: Friday, May 01, 2015 5:25 AM
> > To: dev@kafka.apache.org
> &
nger to identity which configs can be made dynamic and actually doing the
> work to make them so. I think that once we have reasonable agreement on the
> overall picture, we can implement these things piece by piece.
>
> Thanks,
> Aditya
>
> _
the overall picture, we can
implement these things piece by piece.
Thanks,
Aditya
From: Jay Kreps [jay.kr...@gmail.com]
Sent: Friday, May 01, 2015 12:53 PM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-21 Configuration Management
Hey Aditya,
This is a
ed to). Is the AdminChangeNotification a ZK
> notification or a request type exposed by each broker?
>
> Thanks,
> Aditya
>
>
> From: Joe Stein [joe.st...@stealth.ly]
> Sent: Friday, May 01, 2015 5:25 AM
> To: dev@kafka.apache.or
From: Joe Stein [joe.st...@stealth.ly]
Sent: Friday, May 01, 2015 5:25 AM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-21 Configuration Management
Hi Aditya, thanks for the write up and focusing on this piece.
Agreed we need something that we can do
: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-21 Configuration Management
Thanks for starting this discussion, Aditya. Few questions/comments
1. If you change the default values like it's mentioned in the KIP, do you
also overwrite the local config file as part of updating the default val
Thanks for starting this discussion, Aditya. Few questions/comments
1. If you change the default values like it's mentioned in the KIP, do you
also overwrite the local config file as part of updating the default value?
If not, where does the admin look to find the default values, ZK or local
Kafka
Hey Aditya,
This is a great! A couple of comments:
1. Leaving the file config in place is definitely the least disturbance.
But let's really think about getting rid of the files and just have one
config mechanism. There is always a tendency to make everything pluggable
which so often just leads t
es that can be converted slowly.
Thanks,
Aditya
From: Joel Koshy [jjkosh...@gmail.com]
Sent: Thursday, April 30, 2015 5:14 PM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-21 Configuration Management
>1. I have deep concerns about managing configuratio
Hi Aditya, thanks for the write up and focusing on this piece.
Agreed we need something that we can do broker changes dynamically without
rolling restarts.
I think though if every broker is getting changes it with notifications it
is going to limit which configs can be dynamic.
We could never de
>1. I have deep concerns about managing configuration in ZooKeeper.
>First, Producers and Consumers shouldn't depend on ZK at all, this seems
>to add back a dependency we are trying to get away from.
The KIP probably needs to be clarified here - I don't think Aditya was
referring to cl
Hi Aditya,
Thanks for the write-up. I like the idea of marking specific configurations
as dynamic, I think this is just what we need.
Few comments:
1. I have deep concerns about managing configuration in ZooKeeper.
First, Producers and Consumers shouldn't depend on ZK at all, this seems
42 matches
Mail list logo
