Hi all,
I made a change to the KIP to specify sasl.jaas.config property for brokers
separately for each mechanism, rather than together in a single login
context as we do today in the JAAS config file. This will make it easier to
add new mechanisms to a running broker (this is not in scope for thi
Yea, if other commands seem to follow this pattern, I'll update KIP-248 as
well :). Also introducing those arguments in the current ConfigCommand also
makes sense from the migration point of view too as it will be introduced
in 1.1 which makes it somewhat easier for KIP-248.
On Wed, Jan 24, 2018 a
Hi Ismael,
Yes, that makes sense. Looking at the command line options for different
tools, we seem to be using *--command-config *in the commands
that currently talk to the new AdminClient (DelegationTokenCommand,
ConsumerGroupCommand, DeleteRecordsCommand). So perhaps it makes sense to
do the sa
Hi all,
I'd also like to as the community here who were participating the
discussion of KIP-226 to take a look at KIP-248 (that is making
kafka-configs.sh fully function with AdminClient and a Java based
ConfigCommand). It would be much appreciated to get feedback on that as it
plays an important
Hi Rajini,
I think the proposal makes sense. One suggestion: can we just allow the
config to be passed? That is, leave out the properties config for now.
On Tue, Jan 23, 2018 at 3:01 PM, Rajini Sivaram
wrote:
> Since we are running out of time to get the whole ConfigCommand converted
> to using
Since we are running out of time to get the whole ConfigCommand converted
to using the new AdminClient for 1.1.0 (KIP-248), we need a way to enable
ConfigCommand to handle broker config updates (implemented by KIP-226). As
a simple first step, it would make sense to use the existing ConfigCommand
t
Thanks Rajini. Sounds good.
Ismael
On Wed, Jan 10, 2018 at 11:41 AM, Rajini Sivaram
wrote:
> Hi Ismael,
>
> I have updated the KIP to use AES-256 if available and AES-128 otherwise
> for password encryption. Looking at GCM, it looks like GCM is typically
> used with a variable initialization ve
Hi Ismael,
I have updated the KIP to use AES-256 if available and AES-128 otherwise
for password encryption. Looking at GCM, it looks like GCM is typically
used with a variable initialization vector, while we are using a random,
but constant IV per-password. Also, AES/GCM is not supported by Java7
Thanks, Rajini. That makes sense.
regards,
Colin
On Tue, Jan 9, 2018, at 14:38, Rajini Sivaram wrote:
> Hi Colin,
>
> Thank you for reviewing.
>
> Yes, validation is done on the broker, not the client.
>
> All configs from ZooKeeper are processed and any config that could not be
> applied are
Hi Colin,
Thank you for reviewing.
Yes, validation is done on the broker, not the client.
All configs from ZooKeeper are processed and any config that could not be
applied are logged as warnings. This includes any configs that are not
dynamic in the broker version or any configs that are not sup
On Mon, Dec 18, 2017, at 13:40, Jason Gustafson wrote:
> Hi Rajini,
>
> Looking good. Just a few questions.
>
> 1. (Related to Jay's comment) Is the validate() method on Reconfigurable
> necessary? I would have thought we'd validate using the ConfigDef. Do you
> have a use case in mind in which t
Hi Ismael,
Thank you for reviewing the KIP.
*password.encoder.iterations: 2048*: That was a mistake in the doc, changed
to 4096, which is the minimum we use for SCRAM credentials
*password.encoder.key.length: 128: *That is a key size that works with the
default cipher algorithm. Will change if
Hi Rajini,
Quick question (sorry if this was already discussed). How were the
following chosen?
Name: password.encoder.keyfactory.algorithm Type: String Default:
PBKDF2WithHmacSHA512 if available, otherwise PBKDF2WithHmacSHA1 (e.g. Java7)
Name: password.encoder.cipher.algorithm Type: String De
4. I wasn't sure what to do, so I left it in there, so that synonyms is a
self-contained list.
6. We will never store passwords unencrypted, we will forbid them from
being altered if the secret is not configured.
Thank you,
Rajini
On Tue, Dec 19, 2017 at 7:14 PM, Jason Gustafson wrote:
> Hi R
Hi Rajini,
4. Changed is_default to config_source in config_entry in the protocol. It
> will be less confusing that way. The method isDefault() will just
> return configSource
>
Would we still include the active config in the list of synonyms?
6. It is a nice idea to have an automatically gener
Hi Jason,
Thank you!
2. Updated the KIP: Reconfigurable extends Configurable
4. Changed is_default to config_source in config_entry in the protocol. It
will be less confusing that way. The method isDefault() will just
return configSource
== DEFAULT_CONFIG. Have also included the changes to the
Hey Rajini,
Thanks, makes sense. A couple replies:
2. I haven't changed the way Configurable is used. It is still used for
> initial configuration (if the class implements it). Reconfigurable is used
> at the moment only for reconfiguration. The reason I did it that way is
> because for some of t
Hi Jason,
Thank you for reviewing the KIP.
1. ConfigDef is used for validating the type of the value and the
constraints. But I am doing a lot more validation of security configs. For
example, for keystore configuration update, validate() loads the keystore
and if it is an inter-broker listener,
Hi Rajini,
Looking good. Just a few questions.
1. (Related to Jay's comment) Is the validate() method on Reconfigurable
necessary? I would have thought we'd validate using the ConfigDef. Do you
have a use case in mind in which the reconfigurable component only permits
certain reconfigurations?
2.
Hi Jay,
Thank you for reviewing the KIP.
1) Yes, makes sense. I will update the PR. There are some config updates
that may be allowed depending on the context (e.g. some security configs
can be updated for new listeners, but not existing listeners). Perhaps it
is ok to mark them dynamic in the do
Two thoughts on implementation (shouldn't effect the KIP):
1. It might be nice to add a parameter to ConfigDef which says whether a
configuration is dynamically updatable or not so that we can give error
messages if it isn't and also have it reflected in the auto-generated docs.
2. For
Hi Jun,
Thank you!
5. Yes, that makes sense. Agree that we don't want to add protocol changes
to *UpdateMetadataRequest* in this KIP. I have moved the update of
*log.message.format.version* and *inter.broker.protocol.version* to reduce
restarts during upgrade to* Future Work*. We can do this in a
Hi, Rajini,
Thanks for the reply. They all make sense.
5. Got it. Note that currently, only live brokers are registered in ZK.
Another thing is that I am not sure that we want every broker to read all
broker registrations directly from ZK. It's probably better to have the
controller propagate thi
Hi Jun,
Thank you for the review.
1. No, I am hoping to migrate partitions to new threads. We just need to
ensure they don't run concurrently.
2. AdminClient has a validateOnly option for AlterConfigs. Any exceptions
or return value of false from Reconfigurable#validate would fail the
AlterConfi
A couple more things.
6. For the SSL/SASL configurations with the listener prefix, do we need
another level in config_source since it's neither topic nor broker?
7. For include_synonyms in DescribeConfigs, the name makes sense for the
topic level configs. Not sure if it makes sense for other hier
Hi, Rajini,
Thanks for the kip. Looks good overall. A few comments below.
1. "num.replica.fetchers: Affinity of partitions to threads will be
preserved for ordering." Does that mean the new fetcher threads won't be
used until new partitions are added? This may be limiting.
2. I am wondering how
On Tue, Dec 5, 2017, at 06:01, Rajini Sivaram wrote:
> Hi Colin,
>
> KAFKA-5722 already has an owner, so I didn't want to confuse the two
> KIPs. They can be done independently of each other. The goal is to try and
> validate every config to the minimum validation already in the broker for
> the
Hi Colin,
KAFKA-5722 already has an owner, so I didn't want to confuse the two KIPs.
They can be done independently of each other. The goal is to try and
validate every config to the minimum validation already in the broker for
the static configs, but in some cases to a more restrictive level. So
On Tue, Nov 28, 2017, at 14:48, Rajini Sivaram wrote:
> Hi Colin,
>
> Thank you for reviewing the KIP.
>
> *kaka-configs.sh* will be converted to use *AdminClient* under
> KAFKA-5722.
> This is targeted for the next release (1.1.0). Under this KIP, we will
> implement *AdminClient#alterConfigs* f
Thanks for the explanation.
On Fri, Dec 1, 2017 at 10:13 AM, Rajini Sivaram
wrote:
> Hi Ted,
>
> Thank you for the review. */config/brokers/id *contains the persistent
> configuration of broker *. *That will be configured using
> kafka-configs.sh/AdminClient. */brokers/ids/id* contains the ephem
Hi Ted,
Thank you for the review. */config/brokers/id *contains the persistent
configuration of broker *. *That will be configured using
kafka-configs.sh/AdminClient. */brokers/ids/id* contains the ephemeral
metadata registered by broker *.* For broker version, we don't want the
data to outlive th
Thanks for the update.
bq. To enable this, the version of the broker will be added to the JSON
registered by each broker during startup at */brokers/ids/id*
*It seems the path has typo. Should it be:*
*/config/brokers/id*
*Cheers*
On Fri, Dec 1, 2017 at 8:32 AM, Rajini Sivaram
wrote:
> Made o
Made one change to the KIP - I added a *validate* method to the
*Reconfigurable* interface so that we can validate new configs before they
are applied.
A couple of initial implementations:
1. Dynamic updates of keystores:
https://github.com/rajinisivaram/kafka/commit/3071b686973a59a45546e9
Hi all,
If there are no other suggestions or comments, I will start vote next week.
In the meantime, please feel free to review and add any suggestions or
improvements.
Regards,
Rajini
On Wed, Nov 29, 2017 at 11:52 AM, Rajini Sivaram
wrote:
> Hi Jason,
>
> Thanks for reviewing the KIP.
>
> I
Hi Jason,
Thanks for reviewing the KIP.
I hadn't included *inter.broker.protocol.version*, but you have provided a
good reason to do that in order to avoid an additional rolling restart
during upgrade. I had included *log.message.format.version* along with
other default topic configs, but it prob
Hi Rajini,
One quick question I was wondering about is whether this could be used to
update the inter-broker protocol version or the message format version?
Potentially then we'd only need one rolling restart to upgrade the cluster.
I glanced quickly through the uses of this config in the code and
Hi Colin,
Thank you for reviewing the KIP.
*kaka-configs.sh* will be converted to use *AdminClient* under KAFKA-5722.
This is targeted for the next release (1.1.0). Under this KIP, we will
implement *AdminClient#alterConfigs* for the dynamic configs listed in the
KIP. This will include validation
Hi Rajini,
This seems like a nice improvement!
One thing that is a bit concerning is that, if bin/kafka-configs.sh is
used, there is no way for the broker to give feedback or error
messages. The broker can't say "sorry, I can't reconfigure that in that
way." Or even "that configuration propert
Hi Tom,
No, I am not proposing this as a way to configure replication quotas. When
you describe broker configs using AdminClient, you will see all the configs
persisted in /configs/brokers/ in ZooKeeper and this includes
leader.replication.throttled.rate, follower.replication.throttled.rate etc. B
Hi Rajini,
Just to clarify, are you proposing this as a way to configure interbroker
throttling/quotas? I don't think you are, just wanted to check (since
KIP-179 proposes a different mechanism for setting them which supports
their automatic removal).
Cheers,
Tom
On 22 November 2017 at 18:28, R
I have made an update to the KIP to optionally return all the config
synonyms in *DescribeConfigsResponse*. The synonyms are returned in the
order of precedence. AlterConfigsResponse will not be modified by the KIP.
Since many configs already have various overrides (e.g. topic configs with
broker o
Hi Ted,
You can quote the config name, but it is not necessary for deleting a
config since the name doesn't contain any special characters that requires
quoting.
On Mon, Nov 20, 2017 at 9:20 PM, Ted Yu wrote:
> Thanks for the quick response.
>
> It seems the config following --delete-config sho
Thanks for the quick response.
It seems the config following --delete-config should be quoted.
Cheers
On Mon, Nov 20, 2017 at 12:02 PM, Rajini Sivaram
wrote:
> Ted,
>
> Have added an example for --delete-config.
>
> On Mon, Nov 20, 2017 at 7:42 PM, Ted Yu wrote:
>
> > bq. There is a --delete-
Ted,
Have added an example for --delete-config.
On Mon, Nov 20, 2017 at 7:42 PM, Ted Yu wrote:
> bq. There is a --delete-config option
>
> Consider adding a sample with the above option to the KIP.
>
> Thanks
>
> On Mon, Nov 20, 2017 at 11:36 AM, Rajini Sivaram
> wrote:
>
> > Hi Ted,
> >
> > T
bq. There is a --delete-config option
Consider adding a sample with the above option to the KIP.
Thanks
On Mon, Nov 20, 2017 at 11:36 AM, Rajini Sivaram
wrote:
> Hi Ted,
>
> Thank you for reviewing the KIP.
>
> *Would decreasing network/IO threads be supported ?*
> Yes, As described in the KIP
Hi Ted,
Thank you for reviewing the KIP.
*Would decreasing network/IO threads be supported ?*
Yes, As described in the KIP, some connections will be closed if network
thread count is reduced (and reconnections will be processed on remaining
threads)
*What if some keys in configs are not in the S
bq. (e.g. increase network/IO threads)
Would decreasing network/IO threads be supported ?
bq. void reconfigure(Map configs);
What if some keys in configs are not in the Set returned by
reconfigurableConfigs()
? Would exception be thrown ?
If so, please specify which exception would be thrown
Hi, can you help me?
i have a problem, i would like to create an avro schema on schema registry
of kafka that references a type that is declared in a different avro
schema, the places under an example:
First avro schema with reference UserAction
{"namespace": "com.myorg.other",
"type": "record",
Hi all,
I have submitted KIP-226 to enable dynamic reconfiguration of brokers
without restart:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-226+-+Dynamic+Broker+Configuration
The KIP proposes to extend the current dynamic replication quota
configuration for brokers to support dynamic re
49 matches
Mail list logo
