Re: New release branch 3.9

2024-08-30 Thread Colin McCabe
No worries. Thanks for checking. best, Colin On Thu, Aug 29, 2024, at 19:00, Kirk True wrote: > Thanks Chia-Ping! > > Sorry for the noise, everyone else :) > >> On Aug 29, 2024, at 6:36 PM, Chia-Ping Tsai wrote: >> >> hi Kirk >> >> I have updated the

Re: [VOTE] KIP-1042: Support for wildcard when creating new acls

2024-08-30 Thread Colin McCabe
they can't handle that. No authorizer forks (or at least, not for things like this.) best, Colin On Fri, Aug 30, 2024, at 01:32, Claude Warren, Jr wrote: > I made it easier to replace the existing StandardAuthorizerData with a > different implementation in order show the Trie implementa

Re: [DISCUSS] KIP-1081: Graduation Steps for Features

2024-08-30 Thread Colin McCabe
On Mon, Aug 26, 2024, at 10:51, Josep Prat wrote: > Hi Colin, > > Names are in the KIP. Level 1 to 4 are never meant to be used outside of > this discussion. It's my, apparently successful, attempt to focus on what > the levels mean instead of their names: > >>

Re: [ANNOUNCE] New committer: Lianet Magrans

2024-08-30 Thread Colin McCabe
Congratulations, Lianet! best, Colin On Fri, Aug 30, 2024, at 04:46, Igor Soarez wrote: > Congratulations Lianet! > > -- > Igor

Re: [EXT] Re: [VOTE] KIP-1033: Add Kafka Streams exception handler for exceptions occurring during processing

2024-09-03 Thread Colin McCabe
Hi Matthias and Sebastien, This sounds like a very small and low-risk PR for a KIP that is new in 3.9 (KIP-1033). I think we should be OK to cherry-pick it to 3.9, even though we're past code freeze. Please try to get it done this week, though :) best, Colin On Fri, Aug 30, 2024, at

Re: New release branch 3.9

2024-09-03 Thread Colin McCabe
On Fri, Aug 30, 2024, at 11:53, Chia-Ping Tsai wrote: > hi Colin > > I open the https://issues.apache.org/jira/browse/KAFKA-17454 to be a > blocker for 3.9 since the e2e transactions_mixed_versions_test fail on > 3.9 definitely > > We will fix it tomorrow. > > Best

Re: New release branch 3.9

2024-09-03 Thread Colin McCabe
On Mon, Sep 2, 2024, at 20:15, Luke Chen wrote: > Hi Colin, > > I've made KAFKA-17412 <https://issues.apache.org/jira/browse/KAFKA-17412> ( > PR <https://github.com/apache/kafka/pull/17051>) as v3.9.0 blocker for > documenting the unclean leader election behavior

Re: [DISCUSS] KAFKA-17316 and KAFKA-17423

2024-09-03 Thread Colin McCabe
On Tue, Sep 3, 2024, at 01:46, Claude Warren, Jr wrote: > Colin, > > I can see that it makes sense to replace the StandardAuthorizer so that there > are not 2 implementations. However, I think the testing framework should > remain so that in future new improved imp

Re: [DISCUSS] KIP-1081: Graduation Steps for Features

2024-09-03 Thread Colin McCabe
. Otherwise, users will have a hard time to really > try the feature and thus kinda defeats the purpose of level 3. +1 > > > Last: @Colin, yes we eventually need to pick names for the levels. But I > believe it's actually the right way to agree on the "what" first, a

Re: Build Updates for week of Sep 9, 2024

2024-09-14 Thread Colin McCabe
Thanks for working on this, David. best, Colin On Thu, Sep 12, 2024, at 11:16, David Arthur wrote: > A lot has been happening with the GitHub Actions build in the past few > weeks. I thought I would share some updates. > > *Build Statistics* > Now that we have all PRs builds r

Re: [ANNOUNCE] New committer: Jeff Kim

2024-09-14 Thread Colin McCabe
Congrats, Jeff! best, Colin On Sat, Sep 14, 2024, at 04:01, Igor Soarez wrote: > Congratulations Jeff! > > -- > Igor

Re: New release branch 3.9

2024-09-14 Thread Colin McCabe
Thanks, everyone. It looks like the blockers list is empty now. I am running some ducktape tests and if they look good, will create a release candidate for 3.9.0 soon. regards, Colin On Wed, Sep 11, 2024, at 10:01, Matthias J. Sax wrote: > Hi Colin, > > we found another blocker bug:

Re: [VOTE] KIP-1086: Add ability to specify a custom produce request parser.

2024-09-14 Thread Colin McCabe
Yes, I agree. I'm going to leave a -1 just for process reasons. Please let's continue the discussion. best, Colin On Fri, Sep 13, 2024, at 06:55, David Arthur wrote: > Max, > > First off, thanks for the KIP! Looking back at the discussion thread, I > don't feel l

Re: [VOTE] KIP-1064 - Upgrade slf4j to 2.x

2024-09-14 Thread Colin McCabe
cies without having a clear idea of who is using them. We didn't do this earlier and nobody complained. Putting more stuff on the CLASSPATH can lead to problems for people. best, Colin On Wed, Sep 11, 2024, at 00:36, Muralidhar Basani wrote: > Hi Chia, thank you for the vote. > I h

Re: [DISCUSS] KIP-1073 Return inactive observer nodes in DescribeQuorum response

2024-09-14 Thread Colin McCabe
ced brokers, for informational reasons. Probably the command-line tool could print out something like "fenced brokers will not be shown" in the case where it's talking to an old server. Or the tool could have a flag like --show-fenced which fails if it can't be done? best, Co

Re: Contribution to RAT

2024-09-14 Thread Colin McCabe
know if this is something they'd like to include or not (hopefully yes.) cheers, Colin On Thu, Aug 15, 2024, at 04:54, Claude Warren, Jr wrote: > Greetings, > > I have been working on Apache RAT recently and I noticed that Kafka has a > very nice XSLT to convert the Rat output to an HT

Re: [DISCUSS] KIP-574: CLI Dynamic Configuration with file input

2020-03-25 Thread Colin McCabe
I agree with this. I think we'd be better off without the --delete-config-file option. It just seems confusing. best, Colin On Wed, Mar 25, 2020, at 03:48, Rajini Sivaram wrote: > Hi Aneel, > > Thanks for the KIP. As configurations get more complex, the ability to > provide

Re: [DISCUSS] KIP-574: CLI Dynamic Configuration with file input

2020-03-25 Thread Colin McCabe
re right and we should keep it simple for now. best, Colin On Wed, Mar 25, 2020, at 01:24, Kamal Chandraprakash wrote: > STDIN wasn't standard practice in other scripts like > kafka-console-consumer.sh, kafka-console-producer.sh and kafka-acls.sh > in which the props file is

Re: [DISCUSS] KIP-584: Versioning scheme for features

2020-03-25 Thread Colin McCabe
e to STDOUT. cheers, Colin On Tue, Mar 24, 2020, at 17:08, Kowshik Prakasam wrote: > Hi all, > > I've opened KIP-584 <https://issues.apache.org/jira/browse/KIP-584> > which > is intended to provide a versioning scheme for features. I'd like to use > this thread to

Re: [VOTE] KIP-455: Create an Administrative API for Replica Reassignment

2020-03-25 Thread Colin McCabe
. It now cancels reassignments for all the partitions supplied in the JSON file. I think this will be easier to use and less complex than the previously proposed two-step process of generating a cancellation plan and applying it. cheers, Colin On Fri, Oct 25, 2019, at 01:04, Stanislav

Re: [VOTE] KIP-447: Producer scalability for exactly once semantics

2020-03-27 Thread Colin McCabe
ere's a strong case to be made that streams should catch the UnsupportedVersionException and fall back to its old behavior. best, Colin > > Let me know if you have any questions, thanks! > > On Thu, Mar 26, 2020 at 12:45 PM Matthias J. Sax wrote: > > > One more change for KIP-447.

Re: [VOTE] KIP-447: Producer scalability for exactly once semantics

2020-03-27 Thread Colin McCabe
On Fri, Mar 27, 2020, at 18:29, Colin McCabe wrote: > On Fri, Mar 27, 2020, at 16:06, Boyang Chen wrote: > > Hey there, > > > > we would like to address an improvement on the > > *Producer#sendOffsetsToTransaction(offsets, > > groupMetadata) *API. Previously we as

Re: [VOTE] KIP-447: Producer scalability for exactly once semantics

2020-03-29 Thread Colin McCabe
feature flags won't make 2.5, so we need some more short-term solution here. Either we punt this KIP to the next release, or we find some way to make it safe for everyone, not just Streams (possibly by only enabling it for streams?) best, Colin On Sat, Mar 28, 2020, at 01:31, Matthias J.

Re: [VOTE] KIP-447: Producer scalability for exactly once semantics

2020-03-30 Thread Colin McCabe
Thanks, Boyang. That seems reasonable. best, Colin On Mon, Mar 30, 2020, at 13:32, Boyang Chen wrote: > Thanks Guozhang for the summary. If everyone agrees to the strategy here, > which include: > > 1. No affection to 2.5 release, maintaining the same crashing logic for new &g

Re: [DISCUSS] KIP-584: Versioning scheme for features

2020-03-30 Thread Colin McCabe
On Thu, Mar 26, 2020, at 19:24, Kowshik Prakasam wrote: > Hi Colin, > > Thanks for the feedback! I've changed the KIP to address your > suggestions. > Please find below my explanation. Here is a link to KIP 584: > https://cwiki.apache.org/confluence/display/KAFKA/KIP-584%3

Re: [DISCUSS] KIP-574: CLI Dynamic Configuration with file input

2020-03-30 Thread Colin McCabe
"here document" ( https://en.wikipedia.org/wiki/Here_document ) best, Colin On Fri, Mar 27, 2020, at 07:46, Aneel Nazareth wrote: > Update: I have simplified the KIP down to just adding the single new > --add-config-file option. Thanks for your input, everyone! > > On Thu, Ma

Re: [VOTE] KIP-574: CLI Dynamic Configuration with file input

2020-03-30 Thread Colin McCabe
Yeah, I will renew my +1 as well in light of the new discussion. Thanks. Colin On Fri, Mar 27, 2020, at 11:22, Kamal Chandraprakash wrote: > +1 (non-binding). Thanks for the KIP! > > On Fri, Mar 27, 2020 at 9:16 PM Gwen Shapira wrote: > > > Vote is still good. We can alway

Re: [DISCUSS] KIP-584: Versioning scheme for features

2020-04-03 Thread Colin McCabe
extra work on callers (they have to define their own concrete class for each interface just to use the API.) There's also probably no reason to have these classes inherit from each other or have complex type relationships. One more nitpick is that Kafka generally doesn't use "ge

Re: [DISCUSS] KIP-587 Suppress detailed responses for handled exceptions in security-sensitive environments

2020-04-03 Thread Colin McCabe
requires this change? By the way, the same concern applies to log messages. We do not log sensitive information such as passwords to the log4j output. If you know of that happening somewhere, please file a bug so it can be fixed. best, Colin On Fri, Apr 3, 2020, at 12:56, Connor Penhale wrote: >

Re: [DISCUSS] KIP-587 Suppress detailed responses for handled exceptions in security-sensitive environments

2020-04-03 Thread Colin McCabe
Also, if you do find a security issue, the process to follow is here: https://kafka.apache.org/project-security.html . best, Colin On Fri, Apr 3, 2020, at 14:20, Colin McCabe wrote: > Hi Connor, > > If we are putting security-sensitive information into REST responses, > that i

Re: [DISCUSS] KIP-584: Versioning scheme for features

2020-04-04 Thread Colin McCabe
On Fri, Apr 3, 2020, at 20:32, Kowshik Prakasam wrote: > > Colin wrote: > > It would be simpler to just say that a feature flag which doesn't appear > > in the znode is considered to be at version level 0. This will also > > simplify the code a lot, I think, since yo

Re: [DISCUSS] KIP-584: Versioning scheme for features

2020-04-04 Thread Colin McCabe
ght? It's certainly something only system administrators should be doing (KIP-455 also specifies ALTER on CLUSTER) best, Colin > 105. If we change delete to disable, it's better to do this consistently in > request protocol and admin api as well. > > 110. The minVersi

Re: [DISCUSS] KIP-584: Versioning scheme for features

2020-04-06 Thread Colin McCabe
turned out to be unnecessary. best, Colin On Mon, Apr 6, 2020, at 12:06, Jun Rao wrote: > Hi, Kowshik, > > Thanks for the reply. A few more replies below. > > 100.6 You can look for the sentence "This operation requires ALTER on > CLUSTER." in KIP-455. Also, you can

Re: [DISCUSS] KIP-590: Redirect Zookeeper Mutation Protocols to The Controller

2020-04-06 Thread Colin McCabe
all the brokers have.) regards, Colin On Mon, Apr 6, 2020, at 14:37, Sönke Liebau wrote: > Hi Boyang, > > thanks for the KIP. Sounds good overall. > > @Tom: I thought about your remark a little and think that in principle we > can get away without forwarding the principal at

Re: [DISCUSS] (KAFKA-9806) authorize cluster operation when creating internal topics

2020-04-06 Thread Colin McCabe
s own principal rather than the caller's. Basically the equivalent of a doAs block (I forget how we do this exactly, but we do have some way of doing it). best, Colin On Mon, Apr 6, 2020, at 02:56, Paolo Moriello wrote: > Hello everybody, > > I've opened a Jira to fix a bug

Re: [DISCUSS] (KAFKA-9806) authorize cluster operation when creating internal topics

2020-04-07 Thread Colin McCabe
On Tue, Apr 7, 2020, at 08:08, Paolo Moriello wrote: > Hi Colin, > > Thanks for your interest in this. I agree with you, this change could break > compatibility. However, changing the source principal is non trivial in > this case. In fact, here the problem is not in the internal

Re: [DISCUSS] (KAFKA-9806) authorize cluster operation when creating internal topics

2020-04-13 Thread Colin McCabe
On Thu, Apr 9, 2020, at 09:36, Paolo Moriello wrote: > Hi Colin, > > Thanks again for checking this out. > > Indeed you are right, a configuration problem is what leads to > authorization failure (and consequently to the internal topics bug): i.e. > incorrect ACLs configu

Re: [kafka-clients] [VOTE] 2.5.0 RC3

2020-04-13 Thread Colin McCabe
+1 (binding) verified checksums ran unitTest ran check best, Colin On Tue, Apr 7, 2020, at 21:03, David Arthur wrote: > Hello Kafka users, developers and client-developers, > > This is the forth candidate for release of Apache Kafka 2.5.0. > > * TLS 1.3 support (1.2 is now the

Re: [DISCUSS] KIP-587 Suppress detailed responses for handled exceptions in security-sensitive environments

2020-04-15 Thread Colin McCabe
internal implementation in the KIP, but since you mentioned it, it's probably worth considering using the current ExceptionMapper class with a different configuration rather than creating a new one. best, Colin On Mon, Apr 13, 2020, at 09:04, Connor Penhale wrote: > Hi Chris! > >

Re: [DISCUSS] KIP-590: Redirect Zookeeper Mutation Protocols to The Controller

2020-04-15 Thread Colin McCabe
operations. Basically the affected operations are changing ACLs, changing dynamic configurations, and changing quotas. best, Colin On Wed, Apr 15, 2020, at 15:25, Ismael Juma wrote: > Hi Boyang, > > Thanks for the KIP. Have we considered that this reduces availability for > thes

Re: [DISCUSS] KIP-590: Redirect Zookeeper Mutation Protocols to The Controller

2020-04-15 Thread Colin McCabe
htly different semantics, and it simplifies things to avoid worrying about that. We should restrict the use of forwarding to just principals that have CLUSTERACTION on CLUSTER for now, so that only the brokers and superusers can do it. best, Colin On Tue, Apr 14, 2020, at 13:15, Boyang Chen wr

Re: [DISCUSS] KIP-590: Redirect Zookeeper Mutation Protocols to The Controller

2020-04-17 Thread Colin McCabe
c more like num-messages-redirected or something like that which operates at the message level, and can be monitored on each broker. best, Colin On Thu, Apr 16, 2020, at 07:40, David Jacot wrote: > Hi Boyang, > > Thanks for the KIP. Overall, it looks good to me. I really like the >

Re: [DISCUSS] KIP-590: Redirect Zookeeper Mutation Protocols to The Controller

2020-04-17 Thread Colin McCabe
for the bridge release. The bridge release upgrade process relies on the old nodes sending their administrative operations to the controller quorum, not directly to zookeeper. best, Colin > > Note that this is different from things like AlterIsr since these calls are > coming from clie

Re: [DISCUSS] KIP-590: Redirect Zookeeper Mutation Protocols to The Controller

2020-04-17 Thread Colin McCabe
kes an RPC to another broker, in theory the ACLs could be totally screwed up and we could get denied. It clearly means that administrator made a mistake, but we still have to handle the case somehow. best, Colin

Re: [DISCUSS] KIP-590: Redirect Zookeeper Mutation Protocols to The Controller

2020-04-18 Thread Colin McCabe
On Fri, Apr 17, 2020, at 13:11, Ismael Juma wrote: > Hi Colin, > > The read/modify/write is protected by the zk version, right? > > Ismael No, we don't use the ZK version when doing the write to the config znodes. We do for ACLs, I think. This is something that we could f

Re: [DISCUSS] KIP-590: Redirect Zookeeper Mutation Protocols to The Controller

2020-04-22 Thread Colin McCabe
would allow us to get around this in the future. We should think about that so that we have this functionality in the future if it's needed. best, Colin On Wed, Apr 22, 2020, at 11:21, Guozhang Wang wrote: > Hello Gwen, > > The purpose here is for maintaining compatibility ol

Re: [DISCUSS] KIP-590: Redirect Zookeeper Mutation Protocols to The Controller

2020-04-23 Thread Colin McCabe
Hmm... Maybe we need to add some way to serialize and deserialize KafkaPrincipal subclasses to/from string. We could add a method to KafkaPrincipalBuilder#deserialize and a method KafkaPrincipal#serialize, I suppose. best, Colin On Thu, Apr 23, 2020, at 02:14, Tom Bentley wrote: > Hi fo

Re: [VOTE] KIP-584: Versioning scheme for features

2020-04-23 Thread Colin McCabe
versions of the same feature, right? So we can just have one number for current version level, rather than two. At least that's what I was thinking -- let me know if I missed something. best, Colin On Tue, Apr 21, 2020, at 13:01, Dhruvil Shah wrote: > Thanks for the KIP! +1 (non-bind

Re: [VOTE] KIP-584: Versioning scheme for features

2020-04-24 Thread Colin McCabe
r hand, for a new cluster, we do want to enable all the possible features immediately . I was proposing this as a way to distinguish the two cases (since the new cluster will never be started with an old IBP). > Colin MccCabe wrote: > > And now, something a little bit bigger (sorry).

Re: [VOTE] KIP-584: Versioning scheme for features

2020-04-27 Thread Colin McCabe
Thanks, Kowshik. +1 (binding) best, Colin On Sat, Apr 25, 2020, at 13:20, Kowshik Prakasam wrote: > Hi Colin, > > Thanks for the explanation! I agree with you, and I have updated the > KIP. > Here is a link to relevant section: > https://cwiki.apache.org/confluence/displa

Re: [DISCUSS] KIP-601: Configurable socket connection timeout

2020-04-28 Thread Colin McCabe
so, the "Compatibility, Deprecation, and Migration Plan" should say that there is no impact (the section is just blank now) Thanks again for the KIP. This seems like it has been a gap in Kafka's error handling for a while, and it's good to see a proposal to fix it. best, Coli

Re: [DISCUSS] KIP-589 Add API to Update Replica State in Controller

2020-04-28 Thread Colin McCabe
ID_REQUEST covers this case. I'd also suggest "AlterReplicaStates[Request,Response]" as a slightly better name for this RPC. cheers, Colin On Tue, Apr 7, 2020, at 12:43, David Arthur wrote: > Hey everyone, > > I'd like to start the discussion for KIP-589

Re: [DISCUSS] KIP-595: A Raft Protocol for the Metadata Quorum

2020-04-28 Thread Colin McCabe
f there is no other configuration specifying the quorum voter set. If we had a kafka.mkfs command, we wouldn't need this key because we could assume that there was always quorum information stored locally. best, Colin On Thu, Apr 16, 2020, at 16:44, Jason Gustafson wrote: >

[DISCUSS] KIP-604: Remove ZooKeeper Flags from the Administrative Tools

2020-04-30 Thread Colin McCabe
Hi all, I posted a KIP about removing the deprecated --zookeeper flags from our administrative tools. Check it out here: https://cwiki.apache.org/confluence/x/kRARCQ best, Colin

[DISCUSS] KIP-554: Add Broker-side SCRAM Config API

2020-04-30 Thread Colin McCabe
Hi all, I posted a KIP about adding a new SCRAM configuration API on the broker. Check it out here if you get a chance: https://cwiki.apache.org/confluence/x/ihERCQ cheers, Colin

Re: [DISCUSS] KIP-554: Add Broker-side SCRAM Config API

2020-05-01 Thread Colin McCabe
On Fri, May 1, 2020, at 08:35, Aneel Nazareth wrote: > Hi Colin, > > Thanks for the KIP. Is it also in scope to add support for the new API > to the Admin interface and the implementation in KafkaAdminClient? > Hi Aneel, Yes, we will have a Java API. The new Admin API is descr

Re: [DISCUSS] KIP-601: Configurable socket connection timeout

2020-05-04 Thread Colin McCabe
I think it makes sense to set a shorter default. best, Colin On Mon, May 4, 2020, at 09:44, Jose Garcia Sancio wrote: > Thanks for the KIP Cheng, > > > The default value will be 10 seconds. > > I think we should make the default the current behavior. Meaning the > defa

Re: [DISCUSS] KIP-554: Add Broker-side SCRAM Config API

2020-05-04 Thread Colin McCabe
e were to re-purpose AlterConfigs / DescribeConfigs for this. I suppose if we wanted to go down this path we could define a new resource and use DescribeConfigs to describe its keys. But its values would always have to be returned as null by DescribeConfigs, since they would be considered "sensitiv

Re: [DISCUSS] KIP-589 Add API to Update Replica State in Controller

2020-05-04 Thread Colin McCabe
a lot, but certainly only be a fraction of what a full LeaderAndIsrRequest would have. Sorry if I'm repeating stuff you already figured out, but I just wanted to be more clear about this (I found it confusing too until David explained it to me originally...) best, Colin On Sat, May 2, 2020

Re: [DISCUSS] KIP-604: Remove ZooKeeper Flags from the Administrative Tools

2020-05-04 Thread Colin McCabe
Hi Sönke, You're right on both counts. Thanks for the corrections. Thinking about this more, I think we should just remove kafka-preferred-replica-election.sh. It just duplicates kafka-leader-election.sh, and it has been deprecated for some time. I changed the KIP. cheers, Colin O

Re: [DISCUSS] KIP-554: Add Broker-side SCRAM Config API

2020-05-04 Thread Colin McCabe
ility policies, Java types defined that make them easy to use, and APIs that can return partial results rather than needing to do the filtering on the client side. best, Colin On Mon, May 4, 2020, at 14:30, Guozhang Wang wrote: > Got it. > > Besides SCRAM, are there other scenarios th

[DISCUSS] Kafka 3.0

2020-05-04 Thread Colin McCabe
next after 2.6? Are there any other things we should change in the 2.6 -> 3.0 transition? best, Colin

Re: [DISCUSS] KIP-601: Configurable socket connection timeout

2020-05-04 Thread Colin McCabe
to do a lazy socket connection time out. That is, we only check > if > we need to timeout a socket when we consider the corresponding node as a > candidate in the node provider. The NodeProvider is an AdminClient abstraction, right? Why wouldn't we implement a connection setup time

Re: [DISCUSS] Kafka 3.0

2020-05-06 Thread Colin McCabe
On Mon, May 4, 2020, at 17:12, Ryanne Dolan wrote: > Hey Colin, I think we should wait until after KIP-500's "bridge > release" so there is a clean break from Zookeeper after 3.0. The > bridge release by definition is an attempt to not break anything, so > it theoretica

Re: [DISCUSS] Kafka 3.0

2020-05-06 Thread Colin McCabe
. Remember that there are a lot of downstream users who won't migrate off of the --zookeeper flags until they're really gone-- things like k8s integrations, puppet, chef, and ansible integrations, and so on. best, Colin > > > On Mon, May 4, 2020 at 5:13 PM Ryanne Dolan wrote:

Re: [DISCUSS] Kafka 3.0

2020-05-06 Thread Colin McCabe
elease might... IF you are using a deprecated feature. Or sometimes if you are using an old JVM you will need to upgrade to upgrade major releases. best, Colin > > On Tue, May 5, 2020 at 2:34 PM Gwen Shapira wrote: > > > It sounds like the decision to make the next release 3.0 i

Re: [DISCUSS] KIP-554: Add Broker-side SCRAM Config API

2020-05-06 Thread Colin McCabe
On Tue, May 5, 2020, at 08:12, Tom Bentley wrote: > Hi Colin, > > SCRAM is better than SASL PLAIN because it doesn't send the password over > the wire in the clear. Presumably this property is important for some users > who have chosen to use SCRAM. This proposal does send

Re: [DISCUSS] Kafka 3.0

2020-05-07 Thread Colin McCabe
, dropping the zookeeper flags is a step forward for security and encapsulation which also advances KIP-500. Another example is that removing the kafka-preferred-replica-election.sh command removes a duplicate command that has been deprecated for a while. best, Colin > > On Wed, May

Re: [DISCUSS] Kafka 3.0

2020-05-07 Thread Colin McCabe
er 3.0. I think that's OK and in keeping with how we've handled deprecation and removal in the past. It's important for users to have a smooth upgrade path. best, Colin > > Ryanne > > - > > On Wed, May 6, 2020 at 10:52 PM Colin McCabe wrote: > > &g

Re: [DISCUSS] KIP-554: Add Broker-side SCRAM Config API

2020-05-08 Thread Colin McCabe
On Thu, May 7, 2020, at 04:27, Rajini Sivaram wrote: > Hi Colin, > > Thanks for the KIP. A couple of comments below: > > 1) SCRAM password is never sent over the wire today, not by clients, not by > tools. A salted-hashed version of it stored in ZooKeeper is sent over the >

Re: [DISCUSS] KIP-554: Add Broker-side SCRAM Config API

2020-05-08 Thread Colin McCabe
dependency here. best, Colin On Thu, May 7, 2020, at 05:35, Jakub Scholz wrote: > Hi Colin, > > Could you clarify how this fits with KIP-506 which seems to deal with the > same? > > Thanks & Regards > Jakub > > On Fri, May 1, 2020 at 8:18 AM Colin McCabe wrote: &g

Re: [DISCUSS] Kafka 3.0

2020-05-08 Thread Colin McCabe
0. I don't have a good sense of how practical it is to deprecate this now, so I will defer to others here. But the KIP freeze for 2.6 is coming soon, so if we want to make the case, now is the time. best, Colin On Thu, May 7, 2020, at 16:28, Guozhang Wang wrote: > Hey folks, > >

Re: [DISCUSS] Kafka 3.0

2020-05-11 Thread Colin McCabe
atic topic creation, but it seems like it would be a big step, given that it's currently on by default. What about if we changed it to be off by default and deprecated in 3.0? Then, people who hadn't yet changed their workflows could turn it on in 3.0, and we could remove it entire

Re: [DISCUSS] Kafka 3.0

2020-05-11 Thread Colin McCabe
tion protocol is not Raft, it shares many similarities with that protocol. So I think it's a bit unfair to say that it is "catastrophic hubris" to believe we can implement the protocol. best, Colin On Sun, May 10, 2020, at 11:02, Michael K. Edwards wrote: > Yes, I've re

[VOTE]: KIP-604: Remove ZooKeeper Flags from the Administrative Tools

2020-05-12 Thread Colin McCabe
xists in combination with the --bootstrap-server flag. best, Colin

Re: [DISCUSS] KIP-554: Add Broker-side SCRAM Config API

2020-05-14 Thread Colin McCabe
On Tue, May 12, 2020, at 06:43, Tom Bentley wrote: > Hi Colin, > > It's not clear whether users of the Java API would need to supply the salt > and salted password directly, or whether the constructor of ScramCredential > would take the password and perform the hashing itse

Re: [DISCUSS] KIP-554: Add Broker-side SCRAM Config API

2020-05-14 Thread Colin McCabe
Hi Cheng, Good point. I updated the KIP to include the same information that is currently returned. best, Colin On Sun, May 10, 2020, at 22:40, Cheng Tan wrote: > Hi Colin, > > > If I understood correctly, in your design, listScramUsers will return > the mechanism and itera

Re: [VOTE] KIP-608: Expose Kafka Metrics in Authorizer

2020-05-18 Thread Colin McCabe
orizer interface directly. Or better yet, just add this field to AuthorizerServerInfo. The purpose of AuthorizerServerInfo was to make new fields like this easy to add. best, Colin On Fri, May 15, 2020, at 14:21, Zhiguo Huang wrote: > Thanks to everyone for their input. I've incorpora

Re: [VOTE] KIP-608: Expose Kafka Metrics in Authorizer

2020-05-18 Thread Colin McCabe
hey're relying on some Metrics function that got removed, they won't be happy. Providing a plugin API means we have to really commit to maintaining that API for years to come. If we don't want to make that kind of commitment, it could be worth exploring different ways of doing w

Re: [DISCUSS] KIP-573: Enable TLSv1.3 by default

2020-05-18 Thread Colin McCabe
Hmm. It would be good to figure out if we are going to remove this compatibility hack in the next major release of Kafka? In other words, in Kafka 3.0, will we enable TLS 1.3 by default even if the cipher suite is specified? best, Colin On Mon, May 18, 2020, at 09:26, Ismael Juma wrote

Re: [DISCUSS] KIP-601: Configurable socket connection timeout

2020-05-18 Thread Colin McCabe
up trying to reconnect to the same node over and over. I'll add some more comments in the vote thread. best, Colin On Fri, May 15, 2020, at 14:13, Rajini Sivaram wrote: > Hi Cheng, > > I am fine with the rest of the KIP apart from the 10s default. If no one > else has any conc

Re: [VOTE] KIP-601: Configurable socket connection timeout in NetworkClient

2020-05-18 Thread Colin McCabe
very rare) case where connections are taking a long time to connect. best, Colin On Fri, May 15, 2020, at 19:07, Cheng Tan wrote: > Hello developers, > > Big thanks for all the feedbacks. KIP-601 is finalized and ready for a vote. > > https://cwiki.apache.org/confluence/displa

Re: [VOTE] KIP-601: Configurable socket connection timeout in NetworkClient

2020-05-18 Thread Colin McCabe
On Mon, May 18, 2020, at 14:41, Cheng Tan wrote: > Dear Colin, > > > Thanks for the suggestions. > > > For example, if a new node joins the cluster, it will have 0 failed connect > > attempts, whereas the existing nodes will probably have more than 0. So > >

Re: [DISCUSS] KIP-601: Configurable socket connection timeout

2020-05-19 Thread Colin McCabe
On Tue, May 19, 2020, at 03:27, Rajini Sivaram wrote: > Hi Colin, > > I do agree about the `leastLoadedNode` case. My question was about the > other cases where we are connecting to a specific node: fetch requests to > leaders, produce requests to leaders, requests to gro

Re: [VOTE]: KIP-604: Remove ZooKeeper Flags from the Administrative Tools

2020-05-19 Thread Colin McCabe
On Tue, May 19, 2020, at 09:31, Jason Gustafson wrote: > Hi Colin, > > Looks good. I just had one question. It sounds like your intent is to > change kafka-configs.sh so that the --zookeeper flag is only supported for > bootstrapping. I assume in the case of SCRAM that we will

Re: [VOTE]: KIP-604: Remove ZooKeeper Flags from the Administrative Tools

2020-05-21 Thread Colin McCabe
Hi all, With 4 binding +1 votes from Guozhang Wang, Manikumar, Mickael Miason, and Jason Gustafson, and 1 non-binding vote from David Jacot, the vote passes. thanks, all. Colin On Wed, May 20, 2020, at 18:16, Jason Gustafson wrote: > Sounds good. +1 from me. > > On Tue, May 19, 202

Re: [VOTE] KIP-578: Add configuration to limit number of partitions

2020-05-27 Thread Colin McCabe
nts. best, Colin On Wed, May 27, 2020, at 09:48, Harsha Chintalapani wrote: > Thanks for the KIP Gokul. This will be really useful for our use cases as > well. > +1 (binding). > > -Harsha > > > On Tue, May 26, 2020 at 12:33 AM, Gokul Ramanan Subramanian < >

Re: [VOTE] KIP-589: Add API to update Replica state in Controller

2020-05-27 Thread Colin McCabe
ith this RPC, we'll need some additional functionality. Specifically, we'll need the ability to tell the controller to stop putting new partitions on the broker that sent the request. That could be done with a separate request or possibly additional flags on this request. In any case, w

Re: [VOTE] KIP-589: Add API to update Replica state in Controller

2020-05-27 Thread Colin McCabe
Thanks, David. +1 (binding). cheers, Colin On Wed, May 27, 2020, at 18:21, David Arthur wrote: > Colin, thanks for the feedback. Good points. I've updated the KIP with your > suggestions. > > -David > > On Wed, May 27, 2020 at 4:05 PM Colin McCabe wrote: > > &

Re: [VOTE] KIP-601: Configurable socket connection timeout in NetworkClient

2020-06-03 Thread Colin McCabe
Hi Cheng, Thanks for working on this. Looks good. How about "socket.connection.setup.timeout.ms" and "socket.connection.setup.timeout.max.ms" (not connections with an S)? +1 (binding) best, Colin On Wed, Jun 3, 2020, at 06:24, Rajini Sivaram wrote: > Hi Cheng, >

Re: [VOTE] KIP-599: Throttle Create Topic, Create Partition and Delete Topic Operations

2020-06-07 Thread Colin McCabe
e the topic creation but a network problem prevents the response from being sent to the client). But this would still be a very big improvement over what we have now. Sorry for commenting so late on this but I got distracted by some other things. I hope we can figure this one out-- I feel lik

Re: [VOTE] KIP-599: Throttle Create Topic, Create Partition and Delete Topic Operations

2020-06-08 Thread Colin McCabe
On Mon, Jun 8, 2020, at 14:41, Jun Rao wrote: > Hi, Colin, > > Thanks for the comment. You brought up several points. > > 1. Should we set up a per user quota? To me, it does seem we need some sort > of a quota. When the controller runs out of resources, ideally, we only >

Re: ARM/PowerPC builds

2022-08-29 Thread Colin McCabe
Hi Matthew, It seems like you are the only one who has talked about seeing M1-specific failures -- so far, at least. So it would be helpful if you could post what you've seen. best, Colin On Tue, Aug 23, 2022, at 03:05, Matthew Benedict de Detrich wrote: > Sorry I just noticed this e

Re: ARM/PowerPC builds

2022-08-29 Thread Colin McCabe
Hi Amit, I don't see why we need to run a PowerPC build on every test run. It seems excessive, given that most Java code should work on PowerPC without any changes. Why not just run it nightly? best, Colin On Mon, Aug 22, 2022, at 08:05, Amit Ramswaroop Baheti wrote: > I am planning

Re: [DISCUSS] Apache Kafka 3.3.0 Release

2022-08-29 Thread Colin McCabe
will still get some good testing today. best, Colin On Mon, Aug 29, 2022, at 10:12, José Armando García Sancio wrote: > The documentation and protocol links are not working. Looking into it. > > https://kafka.apache.org/33/documentation.html > https://kafka.apache.org/33/protocol.htm

Re: [VOTE] KIP-860: Add client-provided option to guard against unintentional replication factor change during partition reassignments

2022-09-07 Thread Colin McCabe
[1, 1]. The supported range is [0, 0]." the user will be confused about what the problem is. Instead, the exception should mention that the broker does not support AllowReplicationFactorChange. best, Colin On Wed, Sep 7, 2022, at 06:11, David Jacot wrote: > +1 from me. Thanks, Stan! >

Re: [DISCISS] KIP-860: Add client-provided option to guard against unintentional replication factor change during partition reassignments

2022-09-07 Thread Colin McCabe
, you could view this as a fix for KIP-455 :) (in my opinion, at least) best, Colin On Wed, Sep 7, 2022, at 07:07, Ismael Juma wrote: > Thanks for the KIP. Can we explain a bit more why this is an important use > case to address? For example, do we have concrete examples of people > run

Re: [DISCUSS] KIP-865 Metadata Transactions

2022-09-09 Thread Colin McCabe
ncept of "last stable offset" to be the last committed offset that is NOT part of an ongoing transaction? Just a nomenclature suggestion... best, Colin On Fri, Sep 9, 2022, at 06:42, David Arthur wrote: > Hey folks, I'd like to start a discussion on the idea of adding > transa

Re: Problem with Kafka KRaft 3.1.X

2022-09-09 Thread Colin McCabe
Hi Paul, As Keith wrote, it does sound like you are hitting a separate Linux limit like the max mmap count. I'm curious how many partitions you can create if you change that config! best, Colin On Tue, Sep 6, 2022, at 14:02, Keith Paulson wrote: > I've had similar errors cause b

Re: [DISCUSS] KIP-865 Metadata Transactions

2022-09-09 Thread Colin McCabe
Also, it looks like someone already claimed KIP-865, so I'd suggest grabbing a new number. :) Colin On Fri, Sep 9, 2022, at 09:38, Colin McCabe wrote: > Thanks for this KIP, David! > > In the "motivation" section, it might help to give a concrete example > of an ope

<    5   6   7   8   9   10   11   12   13   14   >