Re: [DISCUSS]: KIP-625: Richer encodings for integral-typed protocol fields.

2020-06-19 Thread Colin McCabe
On Tue, Jun 16, 2020, at 07:42, Tom Bentley wrote: > Hi Colin, > > Thanks for taking a look at it. Replies inline... > > On Mon, Jun 15, 2020 at 6:22 PM Colin McCabe wrote: > > > Hi Tom, > > > > It's an interesting idea. Obviously protocol buf

Re: [ANNOUNCE] New committer: Boyang Chen

2020-06-22 Thread Colin McCabe
Congratulations, Boyang! cheers, Colin On Mon, Jun 22, 2020, at 16:26, Guozhang Wang wrote: > The PMC for Apache Kafka has invited Boyang Chen as a committer and we are > pleased to announce that he has accepted! > > Boyang has been active in the Kafka community more than two years

Re: [VOTE] KIP-590: Redirect Zookeeper Mutation Protocols to The Controller

2020-06-22 Thread Colin McCabe
> > > On Fri, Jun 19, 2020 at 3:18 PM Ismael Juma wrote: > > > > > > > Hi Colin, > > > > > > > > The KIP states in the Compatibility section (not Future work): > > > > > > > > "To support the proxy of requests

Re: [VOTE] KIP-627: Expose Trogdor-specific JMX Metrics for Tasks and Agents

2020-06-25 Thread Colin McCabe
+1 (binding). Thanks, Sam. best, Colin On Thu, Jun 25, 2020, at 18:05, Gwen Shapira wrote: > +1 (binding) > > Thank you, Sam. It is great to see Trogdor getting the care it deserves. > > On Mon, Jun 22, 2020, 1:46 PM Sam Pal wrote: > > > Hi all, > > > >

Re: Running system tests on mac

2020-06-30 Thread Colin McCabe
Ducktape runs on Python 2. You can't use it with Python 3, as you are trying to do here. If anyone's interested in porting it to Python 3 it would be a good change. Otherwise, using docker as suggested here seems to be the best way to go. best, Colin On Mon, Jun 29, 2020, at 02

Re: [DISCUSS] KIP-554: Add Broker-side SCRAM Config API

2020-06-30 Thread Colin McCabe
Hi Rajini, OK. Let's remove the encrypted credentials from ListScramUsersResponse and the associated API. I have updated the KIP-- take a look when you get a chance. best, Colin On Fri, May 15, 2020, at 06:54, Rajini Sivaram wrote: > Hi Colin, > > We have used different

Re: [DISCUSS] Include min.insync.replicas in MetadataResponse to make Producer smarter in partitioning events

2020-07-07 Thread Colin McCabe
mber of in-sync replicas, not try to optimize for it. best, Colin On Mon, Jul 6, 2020, at 10:38, Arvin Zheng wrote: > Hi everyone, > > I would like to start the discussion for KIP-637 > https://cwiki.apache.org/confluence/display/KAFKA/KIP-637%3A+Include+min.insync.replicas+in+Me

Re: [VOTE] KIP-621: Deprecate and replace DescribeLogDirsResult.all() and .values()

2020-07-07 Thread Colin McCabe
Thanks, Tom. I tried to think of a better way to do this, but I think you're right that we probably just need different methods. +1 (binding). best, Colin On Mon, Jul 6, 2020, at 01:14, Tom Bentley wrote: > Hi, > > I'd like to start a vote on KIP-621 which is about dep

Re: [DISCUSS] KIP-363

2020-07-07 Thread Colin McCabe
until they are upgraded. best, Colin On Wed, Jul 1, 2020, at 09:21, Tom Bentley wrote: > Hi all, > > Following a suggestion from Colin in the KIP-625 discussion thread, I'd > like to start discussion on a much smaller KIP which proposes to make error > codes and messages

[DISCUSS] KIP-631: The Quorum-based Kafka Controller

2020-07-07 Thread Colin McCabe
Hi all, I posted a KIP about how the quorum-based controller envisioned in KIP-500 will work. Please take a look here: https://cwiki.apache.org/confluence/x/4RV4CQ best, Colin

Re: [DISCUSS] KIP-621: Deprecate and replace DescribeLogDirsResult.all() and .values()

2020-07-08 Thread Colin McCabe
elf, since we'll continue to use it internally. best, Colin On Wed, Jul 8, 2020, at 07:41, Dongjin Lee wrote: > Hi Tom, > > Thanks for taking this issue. I opened a PR for this issue earlier, but > your KIP was submitted first. So I closed my one > <https://cwiki.apache.org

Re: [DISCUSS] KIP-631: The Quorum-based Kafka Controller

2020-07-08 Thread Colin McCabe
On Tue, Jul 7, 2020, at 18:27, Ron Dagostino wrote: > HI Colin. Thanks for the KIP. Here is some feedback and various questions. > Hi Ron, Thanks for taking a look! > "*Controller processes will listen on a separate port from brokers. This > will be true even when the brok

[VOTE] KIP-554: Add Broker-side SCRAM Config API

2020-07-09 Thread Colin McCabe
b0%40%3Cdev.kafka.apache.org%3E best, Colin

Re: [DISCUSS] KIP-631: The Quorum-based Kafka Controller

2020-07-09 Thread Colin McCabe
n both the > records are committed? > Hi Unmesh, Since the active controller is the only node writing to the log, there is no need for any kind of synchronization or access control at the log level. best, Colin > > Thanks, > Unmesh > > On Wed, Jul 8, 2020 at 6:57 AM Ron Dagos

Re: [DISCUSS] KIP-631: The Quorum-based Kafka Controller

2020-07-09 Thread Colin McCabe
On Thu, Jul 9, 2020, at 10:40, Tom Bentley wrote: > Hi Colin, > > Thanks for the detailed KIP. > > > "As described in KIP-500, the controller will store its data in the > > internal __kafka_metadata topic." > > The KIP doesn't really explain the ext

Re: [DISCUSS] KIP-631: The Quorum-based Kafka Controller

2020-07-09 Thread Colin McCabe
Hi Unmesh, Yes, once the last stable offset advanced, we would consider the topic creation to be done, and then we could return success to the client. best, Colin On Thu, Jul 9, 2020, at 19:44, Unmesh Joshi wrote: > It still needs HighWaterMark / LastStableOffset to be advanced by

Re: [DISCUSS] KIP-621: Deprecate and replace DescribeLogDirsResult.all() and .values()

2020-07-09 Thread Colin McCabe
to fix the real problem with either solution. best, Colin On Thu, Jul 9, 2020, at 08:39, Tom Bentley wrote: > Hi Dongjin, > > Thanks for updating the link and sorry that you put effort into writing > your own KIP for this. I was aware of KAFKA-8794, but since the PR was > about jav

Re: [DISCUSS] KIP-621: Deprecate and replace DescribeLogDirsResult.all() and .values()

2020-07-09 Thread Colin McCabe
which is what most tools care about anyway. best, Colin On Thu, Jul 9, 2020, at 23:04, Colin McCabe wrote: > Yeah. The issue with subclassing is that it's a source compatibility > break, although not (I think) a binary compatibility break. I don't > think it's really

Re: [DISCUSS] KIP-405: Kafka Tiered Storage

2020-07-10 Thread Colin McCabe
verwhelmed the system with segments. So overall, I like to understand why we want to store metadata on local disk rather than remote, and what the options are for the future. best, Colin On Thu, Jul 9, 2020, at 09:55, Harsha Chintalapani wrote: > Hi Jun, > Thanks for the r

Re: [DISCUSS] KIP-631: The Quorum-based Kafka Controller

2020-07-12 Thread Colin McCabe
On Fri, Jul 10, 2020, at 18:15, Guozhang Wang wrote: > Hello Colin, > > Thanks for the nice written KIP. A few meta comments: > > 1) We need to talk a bit about broker failure detection: is that piggy > backed with fencing? i.e. should the controller immediately migrate leader

Re: [DISCUSS] KIP-631: The Quorum-based Kafka Controller

2020-07-12 Thread Colin McCabe
eated. best, Colin On Fri, Jul 10, 2020, at 04:12, Unmesh Joshi wrote: > I was thinking that we might need something like multi-operation > <https://issues.apache.org/jira/browse/ZOOKEEPER-965> record in zookeeper > to atomically create topic and partition records when this multi r

Re: [VOTE] KIP-554: Add Broker-side SCRAM Config API

2020-07-13 Thread Colin McCabe
On Fri, Jul 10, 2020, at 10:55, Boyang Chen wrote: > Hey Colin, thanks for the KIP. One question I have about AlterScramUsers > RPC is whether we could consolidate the deletion list and alteration list, > since in response we only have a single list of results. The further > benefit

Re: [VOTE] KIP-554: Add Broker-side SCRAM Config API

2020-07-13 Thread Colin McCabe
Hi David, The API is for clients. Brokers will still listen to ZooKeeper to load the SCRAM information. best, Colin On Mon, Jul 13, 2020, at 08:30, David Arthur wrote: > Thanks for the KIP, Colin. The new RPCs look good to me, just one question: > since we don't return the pa

Re: [VOTE] KIP-554: Add Broker-side SCRAM Config API

2020-07-13 Thread Colin McCabe
Thanks, Boyang. Fixed. best, Colin On Mon, Jul 13, 2020, at 08:43, Boyang Chen wrote: > Thanks for the update Colin. One nit comment to fix the RPC type > for AlterScramUsersRequest as: > "apiKey": 51, > "type": "request", > "name": "A

Re: [VOTE] KIP-554: Add Broker-side SCRAM Config API

2020-07-14 Thread Colin McCabe
On Tue, Jul 14, 2020, at 07:57, Ron Dagostino wrote: > Hi again, Colin. I also just realized a couple of other incompatibilities > with the way kafka-configs works today that prevents --bootstrap-server > from being a drop-in replacement. This may or may not be a hard > require

Re: [VOTE] KIP-554: Add Broker-side SCRAM Config API

2020-07-15 Thread Colin McCabe
On Tue, Jul 14, 2020, at 16:23, Ron Dagostino wrote: > Thanks, Colin. > > DescribeScramUsersResponse returns a list of ScramUser instances, which > makes sense, but then each of the ScramUser instances only has a single > ScramUserMechanismInfo instance. I believe it needs a Lis

Re: [VOTE] KIP-554: Add Broker-side SCRAM Config API

2020-07-15 Thread Colin McCabe
Hi all, Thanks, everyone, for reviewing. Since we made a few changes to the RPCs in the last few days, I'm going to extend the vote until Monday and close it out then if it looks good. best, Colin On Wed, Jul 15, 2020, at 14:47, Colin McCabe wrote: > On Tue, Jul 14, 2020, at 16

Re: [DISCUSS] KIP-631: The Quorum-based Kafka Controller

2020-07-15 Thread Colin McCabe
the controller process, even when it's co-located. The main reason is that the controller has a different endpoint (aka port), since it's on a separate network. This works in an easy and straightforward way with NetworkClient, MetadataResponse, etc. etc. when using separate IDs. When usi

Re: [DISCUSS] KIP-631: The Quorum-based Kafka Controller

2020-07-15 Thread Colin McCabe
On Mon, Jul 13, 2020, at 11:08, Boyang Chen wrote: > Hey Colin, some quick questions, > > 1. I looked around and didn't find a config for broker heartbeat interval, > are we piggy-back on some existing configs? > Good point. I meant to add this, bu

Re: [DISCUSS] KIP-405: Kafka Tiered Storage

2020-07-15 Thread Colin McCabe
on in the Kafka ecosystem. best, Colin On Tue, Jul 14, 2020, at 09:39, Ying Zheng wrote: > Hi Jun, > Hi Colin, > > Satish and I are still discussing some details about how to handle > transactions / producer ids. Satish is going to make some minor changes to > RLMM API and othe

Re: [VOTE] KIP-554: Add Broker-side SCRAM Config API

2020-07-16 Thread Colin McCabe
On Thu, Jul 16, 2020, at 08:06, Ron Dagostino wrote: > Thanks, Colin. The standard "about" message for ThrottleTimeMs seems > to be "The duration in milliseconds for which the request was throttled > due to a quota violation, or zero if the request did not violate any

Re: [VOTE] KIP-554: Add Broker-side SCRAM Config API

2020-07-21 Thread Colin McCabe
Hi all, With binding +1s from Rajini Sivaram, David Arthur, and Boyang Chen, and non-binding +1s from Tom Bentley, the vote passes. Thanks to everyone who commented and helped to improve the proposal, especially Ron Dagostino, David, and Boyang. best, Colin On Thu, Jul 16, 2020, at 16:02

Re: [VOTE] KIP-590: Redirect Zookeeper Mutation Protocols to The Controller

2020-07-28 Thread Colin McCabe
oller. Just to reiterate, the controller network is not accessible to clients, much like the ZK network today. It seems like we can still do what we need to do for KIP-599, but we just can't mute the channel. This just means we need to do the backpressure at a slightly higher level. best, Coli

Re: [VOTE] KIP-590: Redirect Zookeeper Mutation Protocols to The Controller

2020-07-28 Thread Colin McCabe
failed for the > request during forwarding, this indicates an internal error on the broker > cluster security setup.", BrokerAuthorizationFailureException::new); Grammar nitpick: It would be good to have a period between "forwarding" and "this" to avoid a run-on

Re: [DISCUSS] KIP-631: The Quorum-based Kafka Controller

2020-07-29 Thread Colin McCabe
On Thu, Jul 23, 2020, at 23:02, Boyang Chen wrote: > Hey Colin, > > some more questions I have about the proposal: > > 1. We mentioned in the networking section that "The only time when clients > should contact a controller node directly is when they are debugging system

Re: [DISCUSS] KIP-631: The Quorum-based Kafka Controller

2020-08-03 Thread Colin McCabe
On Mon, Jul 27, 2020, at 09:20, Jun Rao wrote: > Hi, Colin, > > Thanks for the KIP. A few comments below. > Hi Jun, Thanks for the review. Sorry that it took me a while to respond. I wanted to carefully re-read the latest version of KIP-595 first as well as the other discussi

Re: [DISCUSS] KIP-631: The Quorum-based Kafka Controller

2020-08-03 Thread Colin McCabe
o the networking layer. > > 2. Local process pauses can still be an issue on the active controller?. ( > https://issues.apache.org/jira/browse/CASSANDRA-9183) > Cassandra has a very different way of handling cluster membership than we do. This would not be an issue in Kafka because

Re: [VOTE] KIP-595: A Raft Protocol for the Metadata Quorum

2020-08-03 Thread Colin McCabe
ot; when we mean "controller." It's really hard to break old habits. :) best, Colin On Mon, Aug 3, 2020, at 11:19, Ben Stopford wrote: > +1 > > On Mon, 3 Aug 2020 at 19:03, Jason Gustafson wrote: > > > Hi All, I'd like to start a vote on this proposal: &

Re: [DISCUSS] KIP-631: The Quorum-based Kafka Controller

2020-08-03 Thread Colin McCabe
On Mon, Aug 3, 2020, at 15:51, Jose Garcia Sancio wrote: > Thanks for the KIP Colin, > > Here is a partial review: > > > 1. Even when a broker and a controller are co-located in the same JVM, they > > must > > have different node IDs > > Why? What problem

Re: [VOTE] KIP-554: Add Broker-side SCRAM Config API

2020-08-03 Thread Colin McCabe
Thanks, Ron. Sounds good. best, Colin On Tue, Jul 28, 2020, at 15:26, Ron Dagostino wrote: > Hi everyone. I just wanted to notify that while implementing this I > discovered that we had declared the ScramMechanism enum to have the values > HMAC_SHA_{256,512} instead of SCRAM_SHA_{256

Re: [VOTE] KIP-595: A Raft Protocol for the Metadata Quorum

2020-08-04 Thread Colin McCabe
On Mon, Aug 3, 2020, at 20:55, Jason Gustafson wrote: > Hi Colin, > > Thanks for the responses. > > > I have a few lingering questions. I still don't like the fact that the > > leader epoch / fetch epoch is 31 bits. What happens when this rolls over? > > C

Re: [DISCUSS] KIP-631: The Quorum-based Kafka Controller

2020-08-10 Thread Colin McCabe
Hi Jose, That'a s good point that I hadn't considered. It's probably worth having a separate leader change message, as you mentioned. Hi Unmesh, Thanks, I'll take a look. best, Colin On Fri, Aug 7, 2020, at 11:56, Jose Garcia Sancio wrote: > Hi Unmesh, > >

Re: [VOTE] KIP-595: A Raft Protocol for the Metadata Quorum

2020-08-16 Thread Colin McCabe
it's probably better to have a separate KIP about adding checksums to some of our small text files, since this issue seems to exist for many of them. best, Colin > > Ismael > > > On Mon, Aug 3, 2020 at 11:03 AM Jason Gustafson wrote: > > > Hi All, I'd li

Re: [VOTE] KIP-554: Add Broker-side SCRAM Config API

2020-08-28 Thread Colin McCabe
legation tokens, SCRAM, etc. etc. I don't think the proposed restriction you mention here is consistent with that. best, Colin

Re: [DISCUSS] KIP-631: The Quorum-based Kafka Controller

2020-08-28 Thread Colin McCabe
Hi all, I'm thinking of calling a vote on KIP-631 on Monday. Let me know if there's any more comments I should address before I start the vote. cheers, Colin On Tue, Aug 11, 2020, at 05:39, Unmesh Joshi wrote: > >>Hi Unmesh, > >>Thanks, I'll take a look. >

Re: [DISCUSS] KIP-631: The Quorum-based Kafka Controller

2020-08-28 Thread Colin McCabe
On Fri, Aug 28, 2020, at 19:36, Unmesh Joshi wrote: > Hi Colin, > > There were a few of questions I had.. Hi Unmesh, Thanks for the response. > > 1. Were my comments on the broker lease implementation (and corresponding > prototype) appropriate and do we need to change the

Re: [DISCUSS] KIP-631: The Quorum-based Kafka Controller

2020-08-31 Thread Colin McCabe
single threaded as well. I think it makes sense for the controller, since otherwise locking becomes very messy. I'm not sure I understand your question about duplicate broker ID detection, though. There's a section in the KIP about this -- is there a detail we should add there?

Re: [VOTE] KIP-554: Add Broker-side SCRAM Config API

2020-09-01 Thread Colin McCabe
Hi Ron, Thanks. We can wait for Rajini's reply to finalize this, but for now I guess that will unblock the PR at least. If we do decide we want the restriction we can do a follow-on PR. It's good to see this API moving forward! best, Colin On Tue, Sep 1, 2020, at 12:55, Ron

Re: [VOTE] KIP-554: Add Broker-side SCRAM Config API

2020-09-02 Thread Colin McCabe
t of scope for this KIP, since the ZooKeeper-based approach we're replacing didn't support fine-grained permissions. So, can we defer this discussion a bit? best, Colin On Wed, Sep 2, 2020, at 13:08, Rajini Sivaram wrote: > Hi Ron, > > Sounds good to me, thank you. > >

Re: [DISCUSS] KIP-631: The Quorum-based Kafka Controller

2020-09-04 Thread Colin McCabe
> Colin wrote: > > The reason for including LeaseStartTimeMs in the request is to ensure > > that the time required to communicate with the controller gets included in > > the lease time. Since requests can potentially be delayed in the network > > for a long time, this

Re: [DISCUSS] KIP-631: The Quorum-based Kafka Controller

2020-09-11 Thread Colin McCabe
ease > expiration interval. Because registration.lease.timeout.ms, is configured > on the controller, the new active controller will extend all the leases by > registration.lease.timeout.ms. I see that it won't need last heartbeat > time. Agreed. best, Colin > > Thanks, > U

[VOTE] KIP-631: The Quorum-based Kafka Controller

2020-09-14 Thread Colin McCabe
rg%3E Please take a look and vote if you can. best, Colin

Re: [DISCUSS] KIP-631: The Quorum-based Kafka Controller

2020-09-17 Thread Colin McCabe
Hi Unmesh, That's a fair point. I have moved the lease duration to the broker heartbeat response. That way lease durations can be changed just be reconfiguring the controllers. best, Colin On Wed, Sep 16, 2020, at 07:40, Unmesh Joshi wrote: > Thanks Colin, the changes look good to

Re: [DISCUSS] KIP-631: The Quorum-based Kafka Controller

2020-09-24 Thread Colin McCabe
On Mon, Sep 21, 2020, at 18:13, Jun Rao wrote: > Hi, Colin, > > Sorry for the late reply. A few more comments below. > Hi Jun, Thanks for taking another look. > > 50. Configurations > 50.1 controller.listeners: It seems that a controller just needs one > listener.

Re: [DISCUSS] KIP-631: The Quorum-based Kafka Controller

2020-09-24 Thread Colin McCabe
On Thu, Sep 24, 2020, at 16:24, Jun Rao wrote: > Hi, Colin, > > Thanks for the reply and the updated KIP. A few more comments below. > Hi Jun, > > 53. It seems that you already incorporated the changes in KIP-516. With > topic ids, we don't need to wait for the

Re: [VOTE] KIP-584: Versioning scheme for features

2020-09-25 Thread Colin McCabe
vels seems fine. However, please don't use a 32-bit epoch here. We deliberately made the epoch 64 bits to avoid overflow problems in the future once ZK is gone. best, Colin > > 3. Introduced a new 'status' field in the '/features' ZK node schema. The > >

Re: [DISCUSS] KIP-631: The Quorum-based Kafka Controller

2020-09-25 Thread Colin McCabe
On Fri, Sep 25, 2020, at 10:17, Jun Rao wrote: > Hi, Colin, > > Thanks for the reply. > > 60. Yes, I think you are right. We probably need the controller id when a > broker starts up. A broker only stores the Raft leader id in the metadata > file. To do the initial fetch t

Re: [VOTE] KIP-590: Redirect Zookeeper Mutation Protocols to The Controller

2020-09-25 Thread Colin McCabe
make IBP a feature flag, as long as it could be changed without doing a second rolling restart. We actually don't want to have too many feature flags, since it blows up the test matrix. best, Colin > > > > Thanks, > > > > > > > > Jun > > > > &g

Re: [DISCUSS] KIP-631: The Quorum-based Kafka Controller

2020-09-28 Thread Colin McCabe
On Fri, Sep 25, 2020, at 17:35, Jun Rao wrote: > Hi, Colin, > > Thanks for the reply. > > 62. Thinking about this more, I am wondering what's our overall strategy > for configs shared between the broker and the controller. For example, both > the broker and the

Re: KIP-919: Allow AdminClient to Talk Directly with the KRaft Controller Quorum

2023-07-25 Thread Colin McCabe
On Tue, Jul 25, 2023, at 05:30, Luke Chen wrote: > Hi Colin, > > Some more comments: > 1. In the KIP, we mentioned "controller heartbeats", but it is not > explained anywhere. > I think "controller heartbeats" = controller registration", is that > cor

Re: [VOTE] KIP-919: Allow AdminClient to Talk Directly with the KRaft Controller Quorum and add Controller Registration

2023-07-26 Thread Colin McCabe
On Wed, Jul 26, 2023, at 07:09, Ron Dagostino wrote: > Thanks, Colin. +1 (binding) from me. > > I will note that Ziming mentioned in the DISCUSS thread that "There is > a mistake that we use `--bootstrap-server` instead of > `--bootstrap-server(s)`, so should we also cha

Re: [VOTE] KIP-919: Allow AdminClient to Talk Directly with the KRaft Controller Quorum and add Controller Registration

2023-07-26 Thread Colin McCabe
Hi all, With binding +1s from Ron, David, Divij, ziming, and Luke, the vote passes. Thanks, everyone. Best, Colin On Wed, Jul 26, 2023, at 16:02, Colin McCabe wrote: > On Wed, Jul 26, 2023, at 07:09, Ron Dagostino wrote: >> Thanks, Colin. +1 (binding) from me. >> >> I

Re: [VOTE] KIP-919: Allow AdminClient to Talk Directly with the KRaft Controller Quorum and add Controller Registration

2023-07-27 Thread Colin McCabe
I would incline towards plural, but I honestly don't feel that strongly about it. I will just change it to --bootstrap-controller for now, to match --bootstrap-server. Perhaps we can discuss this further in a later KIP, if people are inclined... Colin On Thu, Jul 27, 2023, at 07:13

Re: [VOTE] KIP-942: Add Power(ppc64le) support

2023-08-25 Thread Colin McCabe
Thank you for continuing to work on this. One comment. When we discussed this in the DISCUSS thread, we all wanted to run it nightly in branch builder (or possibly weekly). But looking at the KIP, it doesn't seem to have been updated with the results of these discussions. best, Colin O

Re: What are the biggest issues with Apache Kafka?

2023-08-25 Thread Colin McCabe
If you want to get more familiar with the code base, one great way is to convert more integration tests to KRaft mode. :) best, Colin On Thu, Aug 10, 2023, at 17:16, Liam Hodges wrote: > I'm working with a small team of engineers looking to contribute to the > open source tools

Re: [DISCUSS] KIP-966: Eligible Leader Replicas

2023-09-12 Thread Colin McCabe
oth? This needs to be clarified. Also, if we do want to support lookups by UUID, that is another thing that needs to be added to adminclient. In general, I feel like this should also probably be its own KIP since it's fairly complex best, Colin On Thu, Aug 10, 2023, at 15:46, Calvin Liu

Re: [DISCUSS] KIP-966: Eligible Leader Replicas

2023-09-12 Thread Colin McCabe
uring failovers. best, Colin On Tue, Sep 12, 2023, at 14:25, Colin McCabe wrote: > Hi Calvin, > > Thanks for the KIP. I think this is a great improvement. > >> Additional High Watermark advance requirement > > Typo: change "advance" to "advancement" &

Re: [VOTE] KIP-942: Add Power(ppc64le) support

2023-09-12 Thread Colin McCabe
g, and not something we should make even worse. It leads directly to things like big PR backlogs, which we've discussed here many times. best, Colin On Mon, Sep 11, 2023, at 06:50, Vaibhav Nazare wrote: > Hi Colin > > Can we continue the voting process now? > > Thanks > Vaibha

Re: [DISCUSS] KIP-966: Eligible Leader Replicas

2023-09-13 Thread Colin McCabe
On Tue, Sep 12, 2023, at 17:21, Calvin Liu wrote: > Hi Colin > Thanks for the comments! > Hi Calvin, Thanks again for the KIP. One meta-comment: it's usually better to just do a diff on a message spec file or java file if you're including changes to it in the KIP. This is e

Re: [DISCUSS] KIP-966: Eligible Leader Replicas

2023-09-14 Thread Colin McCabe
es to AdminClient that are needed to use DescribeTopicRequest. Keep at it. It's looking better. :) best, Colin On Thu, Sep 14, 2023, at 11:03, Calvin Liu wrote: > Hi Colin > Thanks for the comments! > > I did the following changes > >1. > >Simplified the API sp

Re: [VOTE] KIP-942: Add Power(ppc64le) support

2023-09-14 Thread Colin McCabe
side note: if you have information about what does and does not work on PPC at the moment, maybe consider posting it to the mailing list. It would be interesting to know. best, Colin On Thu, Sep 14, 2023, at 02:14, Vaibhav Nazare wrote: > Hi Divij, > > Thanks for your response. I hav

Re: [DISCUSS] KIP-966: Eligible Leader Replicas

2023-09-15 Thread Colin McCabe
On Thu, Sep 14, 2023, at 22:23, Calvin Liu wrote: > Hi Colin > 1. I think using the new config name is more clear. >a. The unclean leader election is actually removed if unclean > recovery is in use. >b. Using multiple values in unclean.leader.election.enable is >

Re: [DISCUSS] KIP-966: Eligible Leader Replicas

2023-09-19 Thread Colin McCabe
. best, Colin On Mon, Sep 18, 2023, at 15:54, Calvin Liu wrote: > Hi Colin, > Also, can we deprecate unclean.leader.election.enable in 4.0? Before that, > we can have both the config unclean.recovery.strategy and > unclean.leader.election.enable > and using the unclean.recovery.Enabl

Re: [DISCUSS] KIP-966: Eligible Leader Replicas

2023-09-25 Thread Colin McCabe
clearer. Agreed on your other comments -- thanks again for reviewing. best, Colin > 41. unclean.recovery.manager.enabled: I am not sure why we introduce this > additional config. Is it the same as unclean.recovery.strategy=None? > > 42. DescribeTopicResponse.TopicAuthorizedOperations:

Re: [DISCUSS] KIP-966: Eligible Leader Replicas

2023-10-03 Thread Colin McCabe
future extension? > > 56. Config changes are public facing. Could we have a separate section to > document all the config changes? +1. A separate section for this would be good. best, Colin > > Thanks, > > Jun > > On Mon, Sep 25, 2023 at 4:29 PM Calvin Liu > wrote:

Re: [VOTE]KIP-966: Eligible Leader Replicas

2023-10-06 Thread Colin McCabe
irst 1000, plus a (next_topic, next_partition_id) pair. (if there is nothing more to return, next_topic = null.) With those changes I would be +1 best, Colin If the response wasn't long enough, the caller can set On Wed, Oct 4, 2023, at 17:44, Jun Rao wrote: > Hi, Calvin, > > Thanks for the

Re: [DISCUSS] KIP-979: Allow independently stop KRaft controllers or brokers

2023-10-06 Thread Colin McCabe
Not everything is a broker, though. So --node-id seems better. best, Colin On Sun, Oct 1, 2023, at 23:08, Kamal Chandraprakash wrote: > Hi Hailey, > > Thanks for working on this! This is one of the long-standing open issues. > Now, users have to find the PID of the respective Kafk

Re: [VOTE] KIP-858: Handle JBOD broker disk failure in KRaft

2023-10-06 Thread Colin McCabe
e or not. There are a bunch of ways to handle this... we'll have to think about what is the easiest. Writing the data to ZK while still in hybrid mode might even be on the table as an option. best, Colin On Thu, Oct 5, 2023, at 07:03, David Arthur wrote: > Hey, just chiming in regarding the

Re: [VOTE] KIP-858: Handle JBOD broker disk failure in KRaft

2023-10-09 Thread Colin McCabe
On Fri, Oct 6, 2023, at 18:30, Igor Soarez wrote: > Hi Colin, > >> I would call #2 LOST. It was assigned in the past, but we don't know where. >> I see that you called this OFFLINE). This is not really normal... >> it should happen only when we're migrating from

Re: UncleanLeaderElectionsPerSec metric and Raft

2023-10-24 Thread Colin McCabe
ut adding configuration-based unclean leader election as part of the KIP-966 work. best, Colin On Wed, Oct 18, 2023, at 10:27, Neil Buesing wrote: > Development, > > with Raft controllers, is the unclean leader election / sec metric supose > to be available? > > kafka.controll

Re: [DISCUSS] How to detect (and prevent) complex bugs in Kafka?

2023-10-24 Thread Colin McCabe
the answer. Instead, it might be better to take another look at how we're doing buffer pooling to see if we can simplify. Why are we passing a non-thread-safe object between threads in the first place? Should this be documented better, or better yet, avoided? Why not use a thread-local instea

Re: [DISCUSS] Road to Kafka 4.0

2023-10-27 Thread Colin McCabe
several committers and community members are actively working on KIP-858 and we expect it to land in 3.7 as planned. I will remove the 4.0-blocker label from the others since they do not block 4.0. best, Colin On Wed, Oct 11, 2023, at 05:17, Luke Chen wrote: > Hi all, > > While Kafka

Re: [DISCUSS] Road to Kafka 4.0

2023-10-27 Thread Colin McCabe
Also, Luke, even though I removed the blocker tag from some of these, thanks for taking a look at our backlog. That always helps us improve. If you find something that should be fixed for kraft, go ahead and add the "kraft" tag so we can find it more easily. best, Colin On Fri, Oct 27,

Re: [VOTE] KIP-998: Give ProducerConfig(props, doLog) constructor protected access

2023-11-06 Thread Colin McCabe
in authors -- maybe people want this. And it doesn't require the subclassing hack to get access to it. best, Colin On Fri, Nov 3, 2023, at 17:06, Matthias J. Sax wrote: > Sophie reads my mind well, but I also won't object if majority if people > thinks it's desirable to have

Re: [DISCUSS] Road to Kafka 4.0

2023-11-09 Thread Colin McCabe
since we have only a few weeks left before that release happens. best, Colin On Thu, Nov 9, 2023, at 00:20, Anton Agestam wrote: > Hi Luke, > > We have been looking into what switching from ZK to KRaft will mean for > Aiven. > > We heavily depend on an “immutable infrastructu

[DISCUSS] KIP-1001: Add CurrentControllerId Metric

2023-11-09 Thread Colin McCabe
Hi all, I would like to start the discussion on a KIP I wrote about adding a new CurrentControllerIdMetric. See: https://cwiki.apache.org/confluence/display/KAFKA/KIP-1001%3A+Add+CurrentControllerId+Metric best, Colin

Re: [VOTE] KIP-979 Allow independently stop KRaft processes

2023-11-09 Thread Colin McCabe
into the command line in kafka-server-start.sh. Basically add -Dkafka.node.id=1, -Dkafka.node.roles=broker. This would be helpful to people just examining the output of ps. Finally, you state that either command-line option can be given. What happens if both are given? best, Colin On Mon, Oct

Re: [VOTE] KIP-979 Allow independently stop KRaft processes

2023-11-14 Thread Colin McCabe
Thanks, Hailey. +1 (binding) Colin On Mon, Nov 13, 2023, at 15:13, Hailey Ni wrote: > Hi Colin, > > Thank you for your review. I removed the "absolute path need to be > provided" line from the KIP, and will modify the code to get the absolute > path to the config fil

Re: [DISCUSS] Road to Kafka 4.0

2023-11-14 Thread Colin McCabe
On Tue, Nov 14, 2023, at 04:37, Anton Agestam wrote: > Hi Colin, > > Thank you for your thoughtful and comprehensive response. > >> KIP-853 is not a blocker for either 3.7 or 4.0. We discussed this in >> several KIPs that happened this year and last year. The most notable

[VOTE] KIP-1001; CurrentControllerId Metric

2023-11-14 Thread Colin McCabe
Hi all, I'd like to call a vote for KIP-1001: Add CurrentControllerId metric. Take a look here: https://cwiki.apache.org/confluence/x/egyZE best, Colin

Re: [DISCUSS] Road to Kafka 4.0

2023-11-20 Thread Colin McCabe
of people asking for "just one more feature" before they migrate. best, Colin On Mon, Nov 20, 2023, at 04:15, Josep Prat wrote: > Hi all, > > I wanted to share my opinion regarding this topic. I know some > discussions happened some time ago (over a year) but I believ

Re: [DISCUSS] Road to Kafka 4.0

2023-11-21 Thread Colin McCabe
On Tue, Nov 21, 2023, at 03:47, Josep Prat wrote: > Hi Colin, > > I think it's great that Confluent runs KRaft clusters in production, > and it means that it is production ready for Confluent and it's users. > But luckily for Kafka, the community is bigger than this (sel

Re: [VOTE] KIP-1001; CurrentControllerId Metric

2023-11-21 Thread Colin McCabe
Thanks, everyone! With binding +1s from: José Armando García Sancio Jason Gustafson David Arthur David Jacot the KIP passes. regards, Colin On Tue, Nov 21, 2023, at 09:25, José Armando García Sancio wrote: > LGTM. +1 binding. > > On Mon, Nov 20, 2023 at 1:48 PM Jason Gustafson

Re: [VOTE] KIP-896: Remove old client protocol API versions in Kafka 4.0

2023-11-21 Thread Colin McCabe
Hi Ismael, Can we state somewhere that the message.format.version configuration will be gone in 4.0? We only will support one message format version (for now, at least). If we do want more versions later, I don't think we'll want to configure them via a static config. best, Coli

Re: [VOTE] KIP-896: Remove old client protocol API versions in Kafka 4.0

2023-11-21 Thread Colin McCabe
Ah. I forget that KIP-724 not only deprecated, but proposed a removal in 4.0. Great. +1 (binding) for KIP-896 best, Colin On Tue, Nov 21, 2023, at 12:36, Ismael Juma wrote: > Hi Colin, > > That change was proposed and approved via KIP-724: > > https://cwiki.apache.org/confluenc

Re: [DISCUSS] Road to Kafka 4.0

2023-11-22 Thread Colin McCabe
preciate the offer. But, on the KRaft side, I still maintain that nothing is missing except JBOD, which we already have a plan for. best, Colin > Thank you. > Luke > > On Wed, Nov 22, 2023 at 2:54 AM Colin McCabe wrote: > >> On Tue, Nov 21, 2023, at 03:47, Josep Prat wro

Re: [DISCUSS] Road to Kafka 4.0

2023-11-27 Thread Colin McCabe
"what is to stop us from > redrawing it again and again?", wouldn't the suggestion of parallel work > lanes brought up by Josep address this concern? > It's very important not to fragment the community by supporting multiple long-running branch lines. At the end of the d

Re: [DISCUSS] Apache Kafka 3.8.0 release

2024-06-12 Thread Colin McCabe
soon. We still plan on having KIP-853 in 3.8. In fact, it will be the main feature of that release. best, Colin On Wed, Jun 12, 2024, at 01:50, Josep Prat wrote: > Hi all, > > We are now really close to the planned code freeze deadline (today EOD). > According to KIP-1012 [1] we agreed

Re: [DISCUSS] Apache Kafka 3.8.0 release

2024-06-14 Thread Colin McCabe
rather than 3.8. And we can start doing all the normal release stuff. The main blocker JIRA I'm aware of is KAFKA-16946, which is a very simple fix. best, Colin On Fri, Jun 14, 2024, at 03:48, Satish Duggana wrote: > +1 on going with 3.8 release with the existing plan and targeting the &g

[DISCUSS] Apache Kafka 3.9.0 release

2024-06-14 Thread Colin McCabe
confluence/display/KAFKA/Release+Plan+3.9.0 best, Colin

<    3   4   5   6   7   8   9   10   11   12   >