Fail-fast builds?

2018-12-20 Thread David Arthur
In the jenkins.sh file, we have the following comment: "In order to provide faster feedback, the tasks are ordered so that faster tasks are executed in every module before slower tasks (if possible)" but then we proceed to use the Gradle --continue flag. This means PRs won't get notified of prob

Re: Fail-fast builds?

2018-12-21 Thread David Arthur
Since this is a relatively simple change, I went ahead and opened up a PR here https://github.com/apache/kafka/pull/6059 On Fri, Dec 21, 2018 at 2:15 AM Manikumar wrote: > +1 fo the suggestion. > > On Fri, Dec 21, 2018 at 2:38 AM David Arthur wrote: > > > In the jenkins.sh

Re: [VOTE] 2.2.0 RC2

2019-03-19 Thread David Arthur
+1 Validated signatures, and ran through quick-start. Thanks! On Mon, Mar 18, 2019 at 4:00 AM Jakub Scholz wrote: > +1 (non-binding). I used the staged binaries and run some of my tests > against them. All seems to look good to me. > > On Sat, Mar 9, 2019 at 11:56 PM Matthias J. Sax > wrote:

Re: [VOTE] KIP-392: Allow consumers to fetch from the closest replica

2019-03-25 Thread David Arthur
IP: > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-392%3A+Allow+consumers+to+fetch+from+closest+replica > > > . > > > > > > +1 from me (duh) > > > > > > -Jason > > > > > > -- David Arthur

Re: [VOTE] 2.3.0 RC2

2019-06-17 Thread David Arthur
+1 binding Verified signatures, pulled down kafka_2.12-2.3.0 and ran producer/consumer perf test scripts. -David On Mon, Jun 17, 2019 at 1:48 AM Vahid Hashemian wrote: > +1 (non-binding) > > I also verifies signatures, build from source and tested the Quickstart > successfully on the built bin

Re: [VOTE] KIP-480 : Sticky Partitioner

2019-07-18 Thread David Arthur
+1 binding, looks like a nice improvement. Thanks! -David On Wed, Jul 17, 2019 at 6:17 PM Justine Olshan wrote: > Hello all, > > I wanted to let you all know the KIP has been updated. The > ComputedPartition class has been removed in favor of simply returning an > integer to represent the recor

[DISCUSS] KIP-503: deleted topics metric

2019-08-05 Thread David Arthur
Hello all, I'd like to start a discussion for https://cwiki.apache.org/confluence/display/KAFKA/KIP-503%3A+Add+metric+for+number+of+topics+marked+for+deletion Thanks! David

Re: [DISCUSS] KIP-503: deleted topics metric

2019-08-07 Thread David Arthur
ave. LGTM. > > -Harsha > > > > > > On Mon, Aug 05, 2019 at 11:24 AM, David Arthur > > wrote: > > > > > Hello all, I'd like to start a discussion for > > > https://cwiki.apache.org/confluence/display/KAFKA/ > > > KIP-503%3A+Add+metric+for+number+of+topics+marked+for+deletion > > > > > > Thanks! > > > David > > > > > > > > -- > Best, > Stanislav > -- David Arthur

Re: [DISCUSS] KIP-503: deleted topics metric

2019-08-07 Thread David Arthur
Updated the KIP with a count of replicas awaiting deletion. On Wed, Aug 7, 2019 at 9:37 AM David Arthur wrote: > Thanks for the feedback, Stan. That's a good point about the partition > count -- I'll poke around and see if I can surface this value in the > Controller. > &g

Re: [DISCUSS] KIP-503: deleted topics metric

2019-08-08 Thread David Arthur
ineligible for deletion. Would > it make sense to have a metric for this as well? > > -Jason > > On Wed, Aug 7, 2019 at 1:52 PM David Arthur wrote: > > > Updated the KIP with a count of replicas awaiting deletion. > > > > On Wed, Aug 7, 2019 at 9:37 AM David Arth

Re: [DISCUSS] KIP-503: deleted topics metric

2019-08-08 Thread David Arthur
9 at 4:12 PM David Arthur wrote: > > > Yes I think exposing ineligible topics would be useful as well. The > > controller also tracks this ineligible state for replicas. Would that be > > useful to expose as well? > > > > In that case, we'd be up to four new m

Re: [DISCUSS] KIP-503: deleted topics metric

2019-08-13 Thread David Arthur
hu, Aug 8, 2019 at 5:16 PM David Arthur wrote: > > > It looks like topicsIneligibleForDeletion is a subset of > topicsToBeDeleted > > in the controller. > > > > On Thu, Aug 8, 2019 at 11:16 AM Stanislav Kozlovski < > > stanis...@confluent.io> > >

[VOTE] KIP-503: deleted topics metric

2019-08-13 Thread David Arthur
Hello all, I'd like to start the vote on KIP-503 https://cwiki.apache.org/confluence/display/KAFKA/KIP-503%3A+Add+metric+for+number+of+topics+marked+for+deletion Thanks! David

Re: [VOTE] KIP-497: Add inter-broker API to alter ISR

2019-08-14 Thread David Arthur
; Viktor > > > > > > On Wed, Aug 7, 2019 at 8:37 PM Jason Gustafson > > wrote: > > > > > > > Hi All, > > > > > > > > I'd like to start a vote on KIP-497: > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-497%3A+Add+inter-broker+API+to+alter+ISR > > > > . > > > > +1 > > > > from me. > > > > > > > > -Jason > > > > > > > > > > > > -- > -- Guozhang > -- David Arthur

Re: [VOTE] KIP-503: deleted topics metric

2019-08-19 Thread David Arthur
d On Sun, Aug 18, 2019 at 1:22 PM Gwen Shapira wrote: > +1 (binding) > This will be most useful. Thank you. > > On Tue, Aug 13, 2019 at 12:08 PM David Arthur > wrote: > > > > Hello all, > > > > I'd like to start the vote on KIP-503 > > > https:

Re: [DISCUSS] KIP-500: Replace ZooKeeper with a Self-Managed Metadata Quorum

2019-08-19 Thread David Arthur
t; how > > > > > > > > > much of this can be shared with generic Kafka log code, > and how > > > > > much > > > > > > > > should > > > > > > > > > be different. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > In section "Broker Metadata Management", you mention > "This > > > > > request > > > > > > > will > > > > > > > > > > double as a heartbeat, letting the controller know that > the > > > > > broker > > > > > > is > > > > > > > > > > alive". In section "Broker State Machine", you mention > "The > > > > > > > > MetadataFetch > > > > > > > > > > API serves as this registration mechanism". Does this > mean that > > > > > the > > > > > > > > > > MetadataFetch Request will optionally include broker > > > > > configuration > > > > > > > > > > information? > > > > > > > > > > > > > > > > > > I was originally thinking that the MetadataFetchRequest > should > > > > > > include > > > > > > > > > broker configuration information. Thinking about this > more, maybe > > > > > we > > > > > > > > > should just have a special registration RPC that contains > that > > > > > > > > information, > > > > > > > > > to avoid sending it over the wire all the time. > > > > > > > > > > > > > > > > > > > Does this also mean that MetadataFetch request will > result in > > > > > > > > > > a "write"/AppendEntries through the Raft replication > protocol > > > > > > before > > > > > > > > you > > > > > > > > > > can send the associated MetadataFetch Response? > > > > > > > > > > > > > > > > > > I think we should require the broker to be out of the > Offline state > > > > > > > > before > > > > > > > > > allowing it to fetch metadata, yes. So the separate > registration > > > > > RPC > > > > > > > > > should have completed first. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > In section "Broker State", you mention that a broker can > > > > > transition > > > > > > > to > > > > > > > > > > online after it is caught with the metadata. What do you > mean by > > > > > > > this? > > > > > > > > > > Metadata is always changing. How does the broker know > that it is > > > > > > > caught > > > > > > > > > up > > > > > > > > > > since it doesn't participate in the consensus or the > advancement > > > > > of > > > > > > > the > > > > > > > > > > highwatermark? > > > > > > > > > > > > > > > > > > That's a good point. Being "caught up" is somewhat of a > fuzzy > > > > > > concept > > > > > > > > > here, since the brokers do not participate in the metadata > > > > > consensus. > > > > > > > I > > > > > > > > > think ideally we would want to define it in terms of time > ("the > > > > > > broker > > > > > > > > has > > > > > > > > > all the updates from the last 2 minutes", for example.) > We should > > > > > > > spell > > > > > > > > > this out better in the KIP. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > In section "Start the controller quorum nodes", you > mention "Once > > > > > > it > > > > > > > > has > > > > > > > > > > taken over the /controller node, the active controller > will > > > > > proceed > > > > > > > to > > > > > > > > > load > > > > > > > > > > the full state of ZooKeeper. It will write out this > information > > > > > to > > > > > > > the > > > > > > > > > > quorum's metadata storage. After this point, the > metadata quorum > > > > > > > will > > > > > > > > be > > > > > > > > > > the metadata store of record, rather than the data in > ZooKeeper." > > > > > > > > During > > > > > > > > > > this migration do should we expect to have a small period > > > > > > controller > > > > > > > > > > unavailability while the controller replicas this state > to all of > > > > > > the > > > > > > > > > raft > > > > > > > > > > nodes in the controller quorum and we buffer new > controller API > > > > > > > > requests? > > > > > > > > > > > > > > > > > > Yes, the controller would be unavailable during this > time. I don't > > > > > > > think > > > > > > > > > this will be that different from the current period of > > > > > unavailability > > > > > > > > when > > > > > > > > > a new controller starts up and needs to load the full > state from > > > > > ZK. > > > > > > > The > > > > > > > > > main difference is that in this period, we'd have to write > to the > > > > > > > > > controller quorum rather than just to memory. But we > believe this > > > > > > > should > > > > > > > > > be pretty fast. > > > > > > > > > > > > > > > > > > regards, > > > > > > > > > Colin > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks! > > > > > > > > > > -Jose > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- David Arthur

Re: [VOTE] KIP-482: The Kafka Protocol should Support Optional Tagged Fields

2019-09-04 Thread David Arthur
fka+Protocol+should+Support+Optional+Tagged+Fields > > > > Discussion thread here: > > > > https://lists.apache.org/thread.html/ > > cdc801ae886491b73ef7efecac7ef81b24382f8b6b025899ee343f7a@%3Cdev.kafka. > > apache.org%3E > > > > best, > > Colin > > > > -- > > -Jose > > > > > -- David Arthur

[DISCUSS] 2.3.1 Bug Fix Release

2019-09-05 Thread David Arthur
9?jql=project%20%3D%20KAFKA%20AND%20fixVersion%20%3D%202.3.1 And here is the release plan: https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+2.3.1 Thanks! -- David Arthur

[VOTE] 2.3.1 RC0

2019-09-13 Thread David Arthur
Hello Kafka users, developers and client-developers, This is the first candidate for release of Apache Kafka 2.3.1 which includes many bug fixes for Apache Kafka 2.3. Release notes for the 2.3.1 release: https://home.apache.org/~davidarthur/kafka-2.3.1-rc0/RELEASE_NOTES.html *** Please downl

Re: Delivery Status Notification (Failure)

2019-09-16 Thread David Arthur
And here's a passing build for the 2.3 branch https://builds.apache.org/view/All/job/kafka-2.3-jdk8/108/ On Mon, Sep 16, 2019 at 3:46 PM David Arthur wrote: > And here's a passing build for the 2.3 branch > https://builds.apache.org/view/All/job/kafka-2.3-jdk8/108/ > > On

Re: [VOTE] 2.3.1 RC0

2019-09-25 Thread David Arthur
e.org/jira/browse/KAFKA-8896. The impact of > this > bug is that consumer groups cannot commit offsets or rebalance. The patch > should be ready shortly. > > Thanks, > Jason > > > > On Fri, Sep 13, 2019 at 3:53 PM David Arthur > wrote: > > > Hello Kafka u

Re: Vulnerabilities found for jackson-databind-2.9.9.jar and guava-20.0.jar in latest Apache-kafka latest version 2.3.0

2019-09-30 Thread David Arthur
nd guava-24.1.1-jre.jar jars but the > > > apache-kafka version 2.3.0 does not include these new jars. Can you > help > > > me with this? > > > > > > Regards, > > > Namrata Kokate > > > > > > > > -- David Arthur

[VOTE] 2.3.1 RC1

2019-10-04 Thread David Arthur
nt.io/job/system-test-kafka/job/2.3/ Thanks! David Arthur

Re: [kafka-clients] Re: [VOTE] 2.3.1 RC0

2019-10-04 Thread David Arthur
8649 -- it seems to be a > > critical issue because it affects upgrading of Kafka Streams > > applications. We plan to do a PR asap and hope we can include it in > 2.3.1. > > > > > > -Matthias > > > > On 9/25/19 11:57 AM, David Arthur wrot

Re: [VOTE] 2.3.1 RC1

2019-10-06 Thread David Arthur
Passing builds: Unit/integration tests https://builds.apache.org/job/kafka-2.3-jdk8/122/ System tests https://jenkins.confluent.io/job/system-test-kafka/job/2.3/142/ On Fri, Oct 4, 2019 at 9:52 PM David Arthur wrote: > Hello all, we identified a few bugs and a dependency update we wanted

[VOTE] 2.3.1 RC2

2019-10-18 Thread David Arthur
We found a few more critical issues and so have decided to do one more RC for 2.3.1. Please review the release notes: https://home.apache.org/~davidarthur/kafka-2.3.1-rc2/RELEASE_NOTES.html *** Please download, test and vote by Tuesday, October 22, 9pm PDT Kafka's KEYS file containing PGP keys

Re: [VOTE] 2.3.1 RC2

2019-10-22 Thread David Arthur
> > I also looked over the release notes. I see that KAFKA-8950 is included, so > maybe they just need to be refreshed. > > Thanks for running the release! > > -Jason > > On Fri, Oct 18, 2019 at 5:23 AM David Arthur wrote: > > > We found a few more critical issue

Re: [VOTE] 2.3.1 RC2

2019-10-24 Thread David Arthur
cCabe wrote: > > >> +1. I ran the broker, producer, consumer, etc. > > >> > > >> best, > > >> Colin > > >> > > >> On Tue, Oct 22, 2019, at 13:32, Guozhang Wang wrote: > > >>> +1. I've ran the quic

[ANNOUNCE] Apache Kafka 2.3.1

2019-10-24 Thread David Arthur
contributors to this release! A. Sophie Blee-Goldman, Arjun Satish, Bill Bejeck, Bob Barrett, Boyang Chen, Bruno Cadonna, Cheng Pan, Chia-Ping Tsai, Chris Egerton, Chris Stromberger, Colin P. Mccabe, Colin Patrick McCabe, cpettitt-confluent, cwildman, David Arthur, Dhruvil Shah, Greg Harris, Gunnar

Re: [VOTE] KIP-541: Create a fetch.max.bytes configuration for the broker

2019-10-25 Thread David Arthur
luence/x/4g73Bw > > > > > > Discussion thread: > > > > > > https://lists.apache.org/thread.html/9d9dde93a07e1f1fc8d9f182f94f4bda9d016c5e9f3c8541cdc6f53b@%3Cdev.kafka.apache.org%3E > > > > > > cheers, > > > Colin > > > > > > -- David Arthur

Re: Subject: [VOTE] 2.2.2 RC2

2019-11-08 Thread David Arthur
://home.apache.org/~rhauch/kafka-2.2.2-rc2/javadoc/ > >>> > >>> * Tag to be voted upon (off 2.2 branch) is the 2.2.2 tag: > >>> https://github.com/apache/kafka/releases/tag/2.2.2-rc2 > >>> > >>> * Documentation: > >>> https://kafka.apache.org/22/documentation.html > >>> > >>> * Protocol: > >>> https://kafka.apache.org/22/protocol.html > >>> > >>> * Successful Jenkins builds for the 2.2 branch: > >>> Unit/integration tests: > https://builds.apache.org/job/kafka-2.2-jdk8/1/ > >>> System tests: > >>> https://jenkins.confluent.io/job/system-test-kafka/job/2.2/216/ > >>> > >>> /** > >>> > >>> Thanks, > >>> > >>> Randall Hauch > >> > > -- David Arthur

Re: [DISCUSSION] KIP-619: Add internal topic creation support

2020-06-12 Thread David Arthur
Cheng, thanks for the KIP! Can you include some details about how this will work the post-ZK world? For KafkaAdminClient, will we add a new "internal" field to NewTopic, or will we reuse the existing "configs" map. One concern with sticking this new special field in the topic configs is that we c

Re: [VOTE] KIP-554: Add Broker-side SCRAM Config API

2020-07-13 Thread David Arthur
> > > On Thu, Jul 9, 2020 at 8:49 PM Colin McCabe > wrote: > > > > > > > Hi all, > > > > > > > > I'd like to call a vote for KIP-554: Add a broker-side SCRAM > > > configuration > > > > API. The KIP is here: https://cwiki.apache.org/confluence/x/ihERCQ > > > > > > > > The previous discussion thread is here: > > > > > > > > > > > > https://lists.apache.org/thread.html/r69bdc65bdf58f5576944a551ff249d759073ecbf5daa441cff680ab0%40%3Cdev.kafka.apache.org%3E > > > > > > > > best, > > > > Colin > > > > > > > > > > -- David Arthur

Re: [VOTE] KIP-554: Add Broker-side SCRAM Config API

2020-07-13 Thread David Arthur
2020 at 8:38 AM Colin McCabe wrote: > > > > > Hi David, > > > > > > The API is for clients. Brokers will still listen to ZooKeeper to load > > > the SCRAM information. > > > > > > best, > > > Colin > > > > > > &g

Re: [DISCUSS] KIP-649: Dynamic Client Configuration

2020-08-03 Thread David Arthur
beConfigs and > > IncrementalAlterConfig requests going to get routed by the client to > > the different brokers in the cluster? > > > > 7. > > You mentioned that the producer and the consumer will validate the > > keys and values received from the broker through DescribeConfigs. Will > > the ConfigCommand validate any of the keys or values specified in > > --add-config and --delete-config? Will the broker validate any of the > > keys or values received in the IncrementalAlterConfigs? > > > > 8. > > In rejected ideas the KIP says: > > > This might make sense for certain configurations such as acks, but > does not for others such as timeouts. > > > > I don't think it makes sense even for acks since the clients of the > > Java Producer assume that all of the produce messages are sent with > > the same ack value. > > > > -- > > -Jose > > > -- David Arthur

Re: [DISCUSSION] KIP-619: Add internal topic creation support

2020-08-14 Thread David Arthur
Cheng, Can you clarify a bit more what the difference is between regular topics and internal topics (excluding __consumer_offsets and __transaction_state)? Reading your last message, if internal topics (excluding the two) can be created, deleted, produced to, consumed from, added to transactions,

New PR builder Jenkins job

2020-09-01 Thread David Arthur
Following the migration to the new ci-builds.apache.org, our existing PR builder jobs stopped working. This was due to the removal of a github plugin which we relied on. While looking into how to fix this, we decided to take the opportunity to switch over to a declarative Jenkinsfile for the build.

Re: [VOTE] KIP-919: Allow AdminClient to Talk Directly with the KRaft Controller Quorum and add Controller Registration

2023-07-26 Thread David Arthur
Thanks for driving this KIP, Colin! +1 binding -David On Wed, Jul 26, 2023 at 8:58 AM Divij Vaidya wrote: > +1 (binding) > > -- > Divij Vaidya > > > On Wed, Jul 26, 2023 at 2:56 PM ziming deng > wrote: > > > > +1 (binding) from me. > > > > Thanks for the KIP! > > > > -- > > Ziming > > > > > O

Re: Apache Kafka 3.6.0 release

2023-09-05 Thread David Arthur
> > > >> > > > Vaidya < > > > >> > > > > >> > > > > > > > > > > divijvaidy...@gmail.com> > > > >> > > > > >>

Re: Apache Kafka 3.6.0 release

2023-09-06 Thread David Arthur
if it's helpful. Thanks! David On Tue, Sep 5, 2023 at 8:13 PM Satish Duggana wrote: > Hi David, > Thanks for bringing this issue to this thread. > I marked https://issues.apache.org/jira/browse/KAFKA-15435 as a blocker. > > Thanks, > Satish. > > On Tue, 5 Sept 2023 at 2

Re: Apache Kafka 3.6.0 release

2023-09-08 Thread David Arthur
Quick update on my two blockers: KAFKA-15435 is merged to trunk and cherry-picked to 3.6. I have a PR open for KAFKA-15441 and will hopefully get it merged today. -David On Fri, Sep 8, 2023 at 5:26 AM Ivan Yurchenko wrote: > Hi Satish and all, > > I wonder if https://issues.apache.org/jira/brow

Re: Apache Kafka 3.6.0 release

2023-09-11 Thread David Arthur
. > > > > On Fri, 8 Sept 2023 at 19:23, Ismael Juma wrote: > > > > > > Hi Satish, > > > > > > Do you have a sense of when we'll publish RC0? > > > > > > Thanks, > > > Ismael > > > > > > On Fri, Sep 8, 2023 at 6:27

Re: Apache Kafka 3.6.0 release

2023-09-12 Thread David Arthur
s://github.com/apache/kafka/pull/14370. > > > > > > > > I understand if there are concerns about last minute changes to this > > API > > > > and we can hold off if that makes the most sense. > > > > If we take that route, I think we should still keep

Re: [VOTE] 3.6.0 RC0

2023-09-18 Thread David Arthur
a few runs of unit/integration tests. You can see the latest at > https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.6/. We will > continue > running a few more iterations. > System tests: > We will send an update once we have the results. > > Thanks, > Satish. > -- David Arthur

Re: [VOTE] 3.6.0 RC0

2023-09-19 Thread David Arthur
; > > > > satish.dugg...@gmail.com > > > > > > > > > > > > wrote: > > > > > > > > > > > > > Hi Greg, > > > > > > > Thanks for reporting the KafkaConnect issue. I replied to this > issue &g

Re: [DISCUSS] KIP-966: Eligible Leader Replicas

2023-10-03 Thread David Arthur
Calvin, thanks for the KIP! I'm getting up to speed on the discussion. I had a few questions 57. When is the CleanShutdownFile removed? I think it probably happens after registering with the controller, but it would be good to clarify this. 58. Since the broker epoch comes from the controller, w

Re: [VOTE] KIP-858: Handle JBOD broker disk failure in KRaft

2023-10-05 Thread David Arthur
Hey, just chiming in regarding the ZK migration piece. Generally speaking, one of the design goals of the migration was to have minimal changes on the ZK brokers and especially the ZK controller. Since ZK mode is our safe/well-known fallback mode, we wanted to reduce the chances of introducing bug

Re: Apache Kafka 3.6.0 release

2023-10-05 Thread David Arthur
`connector-plugins` endpoint lists > > duplicates > > > > > which may > > > > > > > > > > > > > cause confusion for users, or poor behavior in > > clients. > > > > > > > > > > > >

Re: [DISCUSS] KIP-966: Eligible Leader Replicas

2023-10-11 Thread David Arthur
One thing we should consider is a static config to totally enable/disable the ELR feature. If I understand the KIP correctly, we can effectively disable the unclean recovery by setting the recovery strategy config to "none". This would make development and rollout of this feature a bit smoother. C

Re: [VOTE] KIP-1001; CurrentControllerId Metric

2023-11-20 Thread David Arthur
Thanks Colin, +1 from me -David On Tue, Nov 14, 2023 at 3:53 PM Colin McCabe wrote: > Hi all, > > I'd like to call a vote for KIP-1001: Add CurrentControllerId metric. > > Take a look here: > https://cwiki.apache.org/confluence/x/egyZE > > best, > Colin > -- -David

Re: [DISCUSS] KIP-1062: Introduce Pagination for some requests used by Admin API

2024-07-12 Thread David Arthur
y nice improvement for > administering > >>> large clusters. > >>>>> > >>>>> AS1: Besides topics, the most numerous resources in Kafka clusters in > >>> my experience > >>>>> are consumer groups. Would it be possible to extend the KIP to cover > >>> ListGroups while > >>>>> you’re in here? I’ve heard of clusters with truly vast numbers of > >>> groups. This is also > >>>>> potentially a sign of a misbehaving or poorly written clients. > Getting > >>> a page of groups > >>>>> with a massive ItemsLeftToFetch would be nice. > >>>>> > >>>>> AS2: A tiny nit: The versions for the added fields are incorrect in > >>> some cases. > >>>>> > >>>>> AS3: I don’t quite understand the cursor for > >>> OffsetFetchRequest/Response. > >>>>> It looks like the cursor is (topic, partition), but not group ID. > Does > >>> the cursor > >>>>> apply to all groups in the request, or is group ID missing? > >>>>> > >>>>> AS4: For the remaining request/response pairs, the cursor makes sense > >>> to me, > >>>>> but I do wonder whether `NextCursor` should be at the top level of > the > >>> responses > >>>>> instead, like DescribeTopicPartitionsResponse. > >>>>> > >>>>> Thanks, > >>>>> Andrew > >>>>> > >>>>>> On 27 Jun 2024, at 14:05, Omnia Ibrahim > >>> wrote: > >>>>>> > >>>>>> Hi everyone, I would like to start a discussion thread for KIP-1062 > >>>>>> > >>> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1062%3A+Introduce+Pagination+for+some+requests+used+by+Admin+API > >>>>>> > >>>>>> > >>>>>> Thanks > >>>>>> Omnia > >>> > >>> > >>> > > > > -- David Arthur

Re: [DISCUSS] KIP-1066: Mechanism to cordon brokers and log directories

2024-07-12 Thread David Arthur
; cordoned > > > > > > > > log dir? > > > > > > > > From the current java doc of the interface, it doesn't look > > > right: > > > > > > > > "Get the default directory for new partitions placed in a > given > > > > > > broker." > > > > > > > > > > > > > > > > 4. Currently, if a broker is registered and then go offline. > In > > > this > > > > > > state, > > > > > > > > the controller will still distribute partitions to this > broker. > > > > > > > > So, if now, the broker get startup with "cordoned.log.dirs" > set, > > > what > > > > > > will > > > > > > > > happen? > > > > > > > > Will the newly assigned partitions be created successfully or > > > not? > > > > > > > > > > > > > > > > 5. I think after a log dir get cordoned, we can always > uncordon > > > it, > > > > > > right? > > > > > > > > I think we should mention it in the KIP. > > > > > > > > > > > > > > > > 6. If a broker is startup with "cordoned.log.dirs" set, and > does > > > that > > > > > > mean > > > > > > > > the internal topics partitions (ex: __consumer_offsets) > cannot be > > > > > > created, > > > > > > > > either? > > > > > > > > Also, if this log dir is happen to be the metadata log dir, > what > > > will > > > > > > > > happen to the metadata topic creation? > > > > > > > > > > > > > > > > Thanks. > > > > > > > > Luke > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Jul 9, 2024 at 12:12 AM Mickael Maison < > > > > > > mickael.mai...@gmail.com> > > > > > > > > wrote: > > > > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > > > > > Thanks for taking a look. > > > > > > > > > > > > > > > > > > - Yes you're right, I meant AlterPartitionReassignments. > Fixed. > > > > > > > > > - That's a good idea. I was expecting users to discover > > > cordoned log > > > > > > > > > directories by describing broker configurations. But being > > > able to > > > > > > > > > also get this information when describing log directories > makes > > > > > > sense. > > > > > > > > > I've added that to the KIP. > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > Mickael > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Jul 5, 2024 at 8:05 AM Haruki Okada < > > > ocadar...@gmail.com> > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > > > > > > > Thank you for the KIP. > > > > > > > > > > The motivation sounds make sense to me. > > > > > > > > > > > > > > > > > > > > I have a few questions: > > > > > > > > > > > > > > > > > > > > - [nits] "AlterPartitions request" in Error handling > section > > > is > > > > > > > > > > "AlterPartitionReassignments request" actually, right? > > > > > > > > > > - Don't we need to include cordoned information in > > > DescribeLogDirs > > > > > > > > > response > > > > > > > > > > too? Some tools (e.g. CruiseControl) need to have a way > to > > > know > > > > > > which > > > > > > > > > > broker/log-dirs are cordoned to generate partition > > > reassignment > > > > > > > > proposal. > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > > > > > 2024年7月4日(木) 22:57 Mickael Maison < > mickael.mai...@gmail.com > > > >: > > > > > > > > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > > > > > > > > > I'd like to start a discussion on KIP-1066 that > introduces > > > a > > > > > > > > mechanism > > > > > > > > > > > to cordon log directories and brokers. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1066%3A+Mechanism+to+cordon+brokers+and+log+directories > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > Mickael > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > Okada Haruki > > > > > > > > > > ocadar...@gmail.com > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- David Arthur

Re: [DISCUSS] KIP-1066: Mechanism to cordon brokers and log directories

2024-07-30 Thread David Arthur
gt; > > have nothing to do. It's opt-in as you need to set cordoned.log.dirs > > > on some brokers to get the new behavior. If you don't want it anymore, > > > you should unset cordoned.log.dirs. Can you explain why this would not > > > work? > &g

Re: [DISCUSS] KIP-1062: Introduce Pagination for some requests used by Admin API

2024-07-30 Thread David Arthur
er/txn requests) and operator of the cluster > need them to take a decision. The pagination just provides them with a way > to escape the timeout problem with large clusters. So am not sure adding > throttling during such time would be wise. > Am happy to add section for throttli

[DISCUSS] GitHub CI

2024-08-15 Thread David Arthur
Hey everyone, Over the past several months (years, maybe?) I've tinkered around with GitHub Actions as a possible alternative to Jenkins for Apache Kafka CI. I think it is time to actually give it an earnest try. We have already done some work with GH Actions. Namely the Docker build and the "sta

Re: [DISCUSS] GitHub CI

2024-08-16 Thread David Arthur
is to write back the result of the CI run as a comment > > on > > > the PR itself. I believe not needing to periodically check CI to see if > > the > > > run finished would be a great win. By having CI commenting on the PR > > > everyone watching the PR (author and reviewers) will get notified when > > it's > > > done. > > > > > -- David Arthur

Re: [VOTE] 3.6.1 RC0

2023-12-04 Thread David Arthur
Mickael, I just filed https://issues.apache.org/jira/browse/KAFKA-15968 while investigating a log corruption issue on the controller. I'm still investigating the issue to see how far back this goes, but I think this could be a blocker. Essentially, the bug is that the controller does not treat a

Re: [VOTE] 3.6.1 RC0

2023-12-04 Thread David Arthur
Commit will crash. Considering these two factors, I don't strongly feel like we need to block the release for this fix. -David On Mon, Dec 4, 2023 at 10:49 AM David Arthur wrote: > Mickael, > > I just filed https://issues.apache.org/jira/browse/KAFKA-15968 while > investigating a

Re: Kafka trunk test & build stability

2023-12-26 Thread David Arthur
S2. We’ve looked into this before, and it wasn’t possible at the time with JUnit. We commonly set a timeout on each test class (especially integration tests). It is probably worth looking at this again and seeing if something has changed with JUnit (or our usage of it) that would allow a global tim

Re: [DISCUSS] KIP-1013: Drop broker and tools support for Java 11 in Kafka 4.0 (deprecate in 3.7)

2023-12-26 Thread David Arthur
Thanks, Ismael. I'm +1 on the proposal. Does this KIP essentially replace KIP-750? On Tue, Dec 26, 2023 at 3:57 PM Ismael Juma wrote: > Hi Colin, > > A couple of comments: > > 1. It is true that full support for OpenJDK 11 from Red Hat will end on > October 2024 (extended life support will cont

Re: [VOTE] KIP-1013: Drop broker and tools support for Java 11 in Kafka 4.0 (deprecate in 3.7)

2024-01-08 Thread David Arthur
; > > > > > > > On Tue, Jan 2, 2024 at 4:30 PM Ismael Juma > > > > wrote: > > > > > > > > > > > > > > > Hi all, > > > > > > > > > > > > > > > > I would like to start a vote on KIP-1013. > > > > > > > > > > > > > > > > As stated in the discussion thread, this KIP was proposed > > after the > > > > > KIP > > > > > > > > freeze for Apache Kafka 3.7, but it is purely a documentation > > > > update > > > > > > (if we > > > > > > > > decide to adopt it) and I believe it would serve our users > > best if > > > > we > > > > > > > > communicate the deprecation for removal sooner (i.e. 3.7) > > rather > > > > than > > > > > > later > > > > > > > > (i.e. 3.8). > > > > > > > > > > > > > > > > Please take a look and cast your vote. > > > > > > > > > > > > > > > > Link: > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=284789510 > > > > > > > > > > > > > > > > Ismael > > > > > > > > > > > > > > > > > > > > > > > > > > -- David Arthur

Github build queue

2024-02-09 Thread David Arthur
b stats, we are averaging under 40 commits per week. Assuming those primarily come in on weekdays, that's 8 commits per day. If we just run "gradlew check -x tests" for the merge queue job, I don't think we'd get backlogged. Thoughts? David -- David Arthur

Re: Github build queue

2024-02-09 Thread David Arthur
he validation checks to be > reliable > > (or else the PR won't be merged). Sounds like you are suggesting to skip > > the tests for the merge queue validation. Could we perhaps include the > unit > > tests as well? That would incentivize us to ensure the unit tests are >

Re: Github build queue

2024-02-09 Thread David Arthur
te: > > > > > Hi David, > > > > > > +1 on that strategy. > > > > > > I see several flaky tests that aren't marked with @Tag("integration") > > > or @IntegrationTest, and I think those would make using the unitTest > > &

Re: Github build queue

2024-02-09 Thread David Arthur
> >> like > >> > > > that would not be too hard. > >> > > > > >> > > > Ismael > >> > > > > >> > > > On Fri, Feb 9, 2024 at 9:03 AM Greg Harris > >> >> > > > > >> > > > wrote: >

Re: [DISCUSS] KIP-966: Eligible Leader Replicas

2024-02-23 Thread David Arthur
also want to have the >pagination on huge topics for their partitions. >2. To avoid OOM, we should only fetch the new topics when we need them >and release the used topics. Especially the main use case of looping the >topic list is when the client prints all the topics. &

Re: [DISCUSS] KIP-966: Eligible Leader Replicas

2024-02-28 Thread David Arthur
Andrew/Jose, I like the suggested Flow API. It's also similar to the stream observers in GPRC. I'm not sure we should expose something as complex as the Flow API directly in KafkaAdminClient, but certainly we can provide a similar interface. --- Cancellations: Another thing not yet discussed is h

Re: [DISCUSS] KIP-932: Queues for Kafka

2024-04-04 Thread David Arthur
Andrew, thanks for the KIP! This is a pretty exciting effort. I've finally made it through the KIP, still trying to grok the whole thing. Sorry if some of my questions are basic :) Concepts: 70. Does the Group Coordinator communicate with the Share Coordinator over RPC or directly in-process?

Re: [DISCUSS] KIP-1036: Extend RecordDeserializationException exception

2024-04-18 Thread David Arthur
Hi Fred, thanks for the KIP. Seems like a useful improvement. As others have mentioned, I think we should avoid exposing Record in this way. Using ConsumerRecord seems okay, but maybe not the best fit for this case (for the reasons Matthias gave). Maybe we could create a new container interface

Re: [DISCUSS] GitHub CI

2024-08-22 Thread David Arthur
jira/browse/INFRA-26011 if you are > interested in the details. > > I'm wondering if we also get access to other architectures via GitHub > actions? > > Thanks, > Mickael > > On Fri, Aug 16, 2024 at 6:02 PM David Arthur wrote: > > > > Josep, > >

Re: [DISCUSS] GitHub CI

2024-08-25 Thread David Arthur
-David On Thu, Aug 22, 2024 at 3:04 PM David Arthur wrote: > The Github public runners (which we are using) only offer windows, mac, > and linux (x86_64). It is possible to set up dedicated "self-hosted" > runners for a project (or org) which would allow whatever architectur

Re: [DISCUSS] KIP-1081: Graduation Steps for Features

2024-08-25 Thread David Arthur
t;>>>>> understand the Docker image is level 4. > >>>>>>>>> > >>>>>>>>> Does that make sense? If so I can update the KIP with those > >>>>>> examples. > >>>>>>>>> > >>>>>>>>> Best, > >>>>>>>>> > >>>>>>>>> -- > >>>>>>>>> Josep Prat > >>>>>>>>> Open Source Engineering Director, Aiven > >>>>>>>>> josep.p...@aiven.io | +491715557497 | aiven.io > >>>>>>>>> Aiven Deutschland GmbH > >>>>>>>>> Alexanderufer 3-7, 10117 Berlin > >>>>>>>>> Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen, > >>>>>>>>> Anna Richardson, Kenneth Chen > >>>>>>>>> Amtsgericht Charlottenburg, HRB 209739 B > >>>>>>>>> > >>>>>>>>> On Sun, Aug 18, 2024, 21:46 Chia-Ping Tsai > >>>>>> wrote: > >>>>>>>>> > >>>>>>>>>> hi Josep > >>>>>>>>>> > >>>>>>>>>> Although I didn't join the discussion before, the KIP is > >>>>>> interesting > >>>>>>>> and > >>>>>>>>>> great to me. > >>>>>>>>>> > >>>>>>>>>> one small comment: > >>>>>>>>>> > >>>>>>>>>> Could you please add existent features as an example to each > level > >>>>>>> for > >>>>>>>>> the > >>>>>>>>>> readers who have poor reading (like me) ? For instance, I guess > >>>>>> the > >>>>>>> new > >>>>>>>>>> coordinator is level 3? tiered storage is level 4? > >>>>>>>>>> > >>>>>>>>>> Best, > >>>>>>>>>> Chia-Ping > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> Josep Prat 於 2024年8月19日 週一 > >>>>>> 上午2:13寫道: > >>>>>>>>>> > >>>>>>>>>>> Hi all, > >>>>>>>>>>> I want to start a discussion for KIP-1081: Graduation Steps for > >>>>>>>>> Features. > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>>> > >>>>>> > >>> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1081%3A+Graduation+Steps+for+Features > >>>>>>>>>>> > >>>>>>>>>>> We already had a bit of a discussion here > >>>>>>>>>>> > >>>>>> https://lists.apache.org/thread/5z6rxvs9m0bro5ssmtg8qcgdk40882bz > >>>>>>> and > >>>>>>>>>> that > >>>>>>>>>>> materialized into this KIP. > >>>>>>>>>>> > >>>>>>>>>>> I deliberately defined the graduation steps without giving them > >>>>>> a > >>>>>>>> name, > >>>>>>>>>> so > >>>>>>>>>>> we don't go bike-shedding there. There is a separate section > for > >>>>>>> the > >>>>>>>>>> names > >>>>>>>>>>> of each step. Also an alternative set of names. I'd like to get > >>>>>>> some > >>>>>>>>>>> feedback on the steps, and also on the names for the steps. > >>>>>>>>>>> > >>>>>>>>>>> Looking forward to your opinions, and hopefully only a tiny bit > >>>>>> of > >>>>>>>>>>> bike-shedding :) > >>>>>>>>>>> > >>>>>>>>>>> Best, > >>>>>>>>>>> > >>>>>>>>>>> -- > >>>>>>>>>>> [image: Aiven] <https://www.aiven.io/> > >>>>>>>>>>> > >>>>>>>>>>> *Josep Prat* > >>>>>>>>>>> Open Source Engineering Director, *Aiven* > >>>>>>>>>>> josep.p...@aiven.io | +491715557497 > >>>>>>>>>>> aiven.io <https://www.aiven.io/> | < > >>>>>>>>>> https://www.facebook.com/aivencloud > >>>>>>>>>>>> > >>>>>>>>>>> <https://www.linkedin.com/company/aiven/> < > >>>>>>>>>>> https://twitter.com/aiven_io> > >>>>>>>>>>> *Aiven Deutschland GmbH* > >>>>>>>>>>> Alexanderufer 3-7, 10117 Berlin > >>>>>>>>>>> Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen, > >>>>>>>>>>> Anna Richardson, Kenneth Chen > >>>>>>>>>>> Amtsgericht Charlottenburg, HRB 209739 B > >>>>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> -- > >>>>>>> [image: Aiven] <https://www.aiven.io/> > >>>>>>> > >>>>>>> *Josep Prat* > >>>>>>> Open Source Engineering Director, *Aiven* > >>>>>>> josep.p...@aiven.io | +491715557497 > >>>>>>> aiven.io <https://www.aiven.io/> | < > >>>>>> https://www.facebook.com/aivencloud > >>>>>>>> > >>>>>>> <https://www.linkedin.com/company/aiven/> < > >>>>>>> https://twitter.com/aiven_io> > >>>>>>> *Aiven Deutschland GmbH* > >>>>>>> Alexanderufer 3-7, 10117 Berlin > >>>>>>> Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen, > >>>>>>> Anna Richardson, Kenneth Chen > >>>>>>> Amtsgericht Charlottenburg, HRB 209739 B > >>>>>>> > >>>>>> > >>>>> > >>>>> > >>>>> -- > >>>>> [image: Aiven] <https://www.aiven.io/> > >>>>> > >>>>> *Josep Prat* > >>>>> Open Source Engineering Director, *Aiven* > >>>>> josep.p...@aiven.io | +491715557497 > >>>>> aiven.io <https://www.aiven.io/> | > >>>>> <https://www.facebook.com/aivencloud> > >>>>> <https://www.linkedin.com/company/aiven/> < > >>> https://twitter.com/aiven_io> > >>>>> *Aiven Deutschland GmbH* > >>>>> Alexanderufer 3-7, 10117 Berlin > >>>>> Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen, > >>>>> Anna Richardson, Kenneth Chen > >>>>> Amtsgericht Charlottenburg, HRB 209739 B > >>>>> > >>>> > >>>> > >>> > >> > >> > >> -- > >> [image: Aiven] <https://www.aiven.io/> > >> > >> *Josep Prat* > >> Open Source Engineering Director, *Aiven* > >> josep.p...@aiven.io | +491715557497 > >> aiven.io <https://www.aiven.io/> | < > https://www.facebook.com/aivencloud> > >> <https://www.linkedin.com/company/aiven/> < > https://twitter.com/aiven_io> > >> *Aiven Deutschland GmbH* > >> Alexanderufer 3-7, 10117 Berlin > >> Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen, > >> Anna Richardson, Kenneth Chen > >> Amtsgericht Charlottenburg, HRB 209739 B > -- David Arthur

Re: [DISCUSS] KIP-1081: Graduation Steps for Features

2024-08-27 Thread David Arthur
s that levels can only be > > >> changed in minors and majors, this means that if we don't say anything > > >> else, the minimum "baking time" would be 1 minor release. This is the > > >> technical lower limit. We could mention that

Re: [ANNOUNCE] New committer: Lianet Magrans

2024-08-28 Thread David Arthur
gt; > > > > > > > > > > -- > > [image: Aiven] <https://www.aiven.io> > > > > *Josep Prat* > > Open Source Engineering Director, *Aiven* > > josep.p...@aiven.io | +491715557497 > > aiven.io <https://www.aiven.io> |

Re: [DISCUSS] GitHub CI

2024-09-04 Thread David Arthur
ours. Next steps I'd like to take 1) Fully enable the GH workflows for all PRs (not just ones with gh- prefix) 2) Continue investigating the build cache ( https://issues.apache.org/jira/browse/KAFKA-17479) 3) Prioritize fixes for the worst flaky tests 4) Identify tests which are causing bu

Re: [ANNOUNCE] New committer: Jeff Kim

2024-09-09 Thread David Arthur
Yash Mayya > >>>>>> wrote: > >>>>>>>>>> > >>>>>>>>>>> Congratulations Jeff! > >>>>>>>>>>> > >>>>>>>>>>> On Mon, 9 Sept, 2024, 12:13 David Jacot, > >>>>>> wrote: > >>>>>>>>>>> > >>>>>>>>>>>> Hi all, > >>>>>>>>>>>> > >>>>>>>>>>>> The PMC of Apache Kafka is pleased to announce a new Kafka > >>>>>>>> committer, > >>>>>>>>>>> Jeff > >>>>>>>>>>>> Kim. > >>>>>>>>>>>> > >>>>>>>>>>>> Jeff has been a Kafka contributor since May 2020. In addition > >>>>>> to > >>>>>>>>> being > >>>>>>>>>>>> a regular contributor and reviewer, he has made significant > >>>>>>>>>>>> contributions to the next generation of the consumer rebalance > >>>>>>>>>>>> protocol (KIP-848) and to the new group coordinator. He > >>>>>> authored > >>>>>>>>>>>> KIP-915 which improved how coordinators can be downgraded. He > >>>>>>> also > >>>>>>>>>>>> contributed multiple fixes/improvements to the fetch from > >>>>>>> follower > >>>>>>>>>>>> feature. > >>>>>>>>>>>> > >>>>>>>>>>>> Congratulations, Jeff! > >>>>>>>>>>>> > >>>>>>>>>>>> Thanks, > >>>>>>>>>>>> David (on behalf of the Apache Kafka PMC) > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>>> > >>>>>> > >>>>> > >>> > > > > > > > -- David Arthur

Re: [DISCUSS] Regarding Old PRs

2024-09-11 Thread David Arthur
or actions/stale: https://github.com/actions/stale Cheers, David A On Sat, Jun 10, 2023 at 2:53 AM David Jacot wrote: > Thanks, David. I left a few comments in the PR. > > -David > > Le ven. 9 juin 2023 à 15:31, David Arthur .invalid> > a écrit : > > > Hey all, I just

Build Updates for week of Sep 9, 2024

2024-09-12 Thread David Arthur
A lot has been happening with the GitHub Actions build in the past few weeks. I thought I would share some updates. *Build Statistics* Now that we have all PRs builds running the test suite (see note below), we can do a better comparison between GH and Jenkins Github Actions Successful trunk buil

Re: [VOTE] KIP-1086: Add ability to specify a custom produce request parser.

2024-09-13 Thread David Arthur
he.org/confluence/pages/viewpage.action?pageId=318606528 > > Discussion: > https://lists.apache.org/thread/wtgt9jql43qmfsmvqcz0y1phc2n08440 > > Thank you, > > Max > > > -- David Arthur

Re: [VOTE] 2.5.0 RC2

2020-03-30 Thread David Arthur
impact of this bug is that it could throw fatal > > exception > > > and kill a stream thread on Kafka Streams level. It could also create a > > > crashing scenario for plain Kafka Consumer users as well as the > exception > > > will be thrown all the way up. > &g

[DISCUSS] KIP-589 Add API to Update Replica State in Controller

2020-04-07 Thread David Arthur
Hey everyone, I'd like to start the discussion for KIP-589, part of the KIP-500 effort https://cwiki.apache.org/confluence/display/KAFKA/KIP-589+Add+API+to+update+Replica+state+in+Controller This is a proposal to use a new RPC instead of ZooKeeper for notifying the controller of an offline repli

[VOTE] 2.5.0 RC3

2020-04-07 Thread David Arthur
Hello Kafka users, developers and client-developers, This is the forth candidate for release of Apache Kafka 2.5.0. * TLS 1.3 support (1.2 is now the default) * Co-groups for Kafka Streams * Incremental rebalance for Kafka Consumer * New metrics for better operational insight * Upgrade Zookeeper

Re: [VOTE] 2.5.0 RC3

2020-04-08 Thread David Arthur
Passing Jenkins build on 2.5 branch: https://builds.apache.org/job/kafka-2.5-jdk8/90/ On Wed, Apr 8, 2020 at 12:03 AM David Arthur wrote: > Hello Kafka users, developers and client-developers, > > This is the forth candidate for release of Apache Kafka 2.5.0. > > * TLS 1.3 supp

[RESULTS] [VOTE] 2.5.0 RC3

2020-04-14 Thread David Arthur
ksums >> ran unitTest >> ran check >> >> best, >> Colin >> >> On Tue, Apr 7, 2020, at 21:03, David Arthur wrote: >> > Hello Kafka users, developers and client-developers, >> > >> > This is the forth candidate for release of Apache Ka

[ANNOUNCE] Apache Kafka 2.5.0

2020-04-15 Thread David Arthur
Stromberger, Colin P. Mccabe, Colin Patrick McCabe, commandini, Cyrus Vafadari, Dae-Ho Kim, David Arthur, David Jacot, David Kim, David Mao, dengziming, Dhruvil Shah, Edoardo Comar, Eduardo Pinto, Fábio Silva, gkomissarov, Grant Henke, Greg Harris, Gunnar Morling, Guozhang Wang, Harsha Laxman

Re: [ANNOUNCE] Apache Kafka 2.5.0

2020-04-16 Thread David Arthur
I've just published a blog post highlighting many of the improvements that landed with 2.5.0. https://blogs.apache.org/kafka/entry/what-s-new-in-apache2 -David On Wed, Apr 15, 2020 at 4:15 PM David Arthur wrote: > The Apache Kafka community is pleased to announce the release fo

Re: [DISCUSS] KIP-589 Add API to Update Replica State in Controller

2020-05-01 Thread David Arthur
d and > some > > can't. > > > > Does EventType need to be an Int32? For enums, we usually use the > > smallest reasonable type, which would be Int8 here. We can always change > > the schema later if needed. UNKNOWN_REPLICA_EVENT_TYPE seems unnecessary &g

Re: [DISCUSS] KIP-589 Add API to Update Replica State in Controller

2020-05-18 Thread David Arthur
7;t strictly required. In the "RPC semantics" > > section, I > > > > mention that if no topics are present in the RPC request, then all > > topics > > > > on the broker are implied. And if a topic is given with no > partitions, > > all > &g

Re: [DISCUSS] KIP-589 Add API to Update Replica State in Controller

2020-05-19 Thread David Arthur
is is > outside the scope of this KIP, but might be worth mentioning as "Future > work" somewhere. > > Thanks, > Jason > > > On Mon, May 18, 2020 at 10:09 AM David Arthur wrote: > > > I've updated the KIP with the feedback from this discussion > >

[VOTE] KIP-589: Add API to update Replica state in Controller

2020-05-20 Thread David Arthur
Hello, all. I'd like to start the vote for KIP-589 which proposes to add a new AlterReplicaState RPC. https://cwiki.apache.org/confluence/display/KAFKA/KIP-589+Add+API+to+update+Replica+state+in+Controller Cheers, David

Re: [VOTE] KIP-589: Add API to update Replica state in Controller

2020-05-27 Thread David Arthur
place > ControlledShutdownRequest with this RPC, we'll need some additional > functionality. Specifically, we'll need the ability to tell the controller > to stop putting new partitions on the broker that sent the request. That > could be done with a separate request or possi

Re: [VOTE] KIP-589: Add API to update Replica state in Controller

2020-06-03 Thread David Arthur
get > > broker epoch since it is updated as ZKversion today. I believe we would > > have this discussion in a different KIP though. > > > > > > Guozhang > > > > On Wed, May 27, 2020 at 8:26 PM Colin McCabe wrote: > > > > > Thanks, David. +

[DISCUSS] KIP-865 Metadata Transactions

2022-09-09 Thread David Arthur
Hey folks, I'd like to start a discussion on the idea of adding transactions in the KRaft controller. This will allow us to overcome the current limitation of atomic batch sizes in Raft which lets us do things like create topics with a huge number of partitions. https://cwiki.apache.org/confluence

[DISCUSS] KIP-868 Metadata Transactions (new thread)

2022-09-09 Thread David Arthur
Starting a new thread to avoid issues with mail client threading. Original thread follows: Hey folks, I'd like to start a discussion on the idea of adding transactions in the KRaft controller. This will allow us to overcome the current limitation of atomic batch sizes in Raft which lets us do thi

Re: [DISCUSS] KIP-865 Metadata Transactions

2022-09-09 Thread David Arthur
accumulating big lists of records which would then have to be applied > > when the transaction completed, rather than building up a MetadataDelta > > (or updating the controller state) incrementally. > > > > Maybe we want to introduce the concept of "last stable offset" to b

Re: [DISCUSS] KIP-868 Metadata Transactions (new thread)

2022-09-19 Thread David Arthur
KIP! > It's a light-weight transactional proposal for single producer, cool! > +1 for it! > > Luke > > > On Sat, Sep 10, 2022 at 1:14 AM David Arthur > wrote: > > > Starting a new thread to avoid issues with mail client threading. > > > > Original thr

Re: [DISCUSS] Apache Kafka 3.3.0 Release

2022-09-19 Thread David Arthur
Hey folks, José has asked me to help push the release along this week while he's out of the office. -David On Tue, Aug 30, 2022 at 12:01 PM José Armando García Sancio wrote: > Thanks Artem and Colin for identifying and fixing the issues > KAFKA-14156 and KAFKA-14187. I have marked both of them

[VOTE] 3.3.0 RC2

2022-09-20 Thread David Arthur
/kafka/releases/tag/3.3.0-rc2 * Documentation: https://kafka.apache.org/33/documentation.html * Protocol: https://kafka.apache.org/33/protocol.html Successful Jenkins builds to follow in a future update to this email. Thanks! David Arthur

Re: [DISCUSS] KIP-868 Metadata Transactions (new thread)

2022-09-21 Thread David Arthur
; > 3. The record name seems vague, and we already have a `EndTransactionMarker` > class in `org.apache.kafka.common.record`, how about > `BeginMetadataTransactionRecord`? > > - - > Best, > Ziming > > > On Sep 10, 2022, at 1:13 AM, David Arthur wrote: > > > >

Re: [kafka-clients] Re: [VOTE] 3.3.0 RC2

2022-09-22 Thread David Arthur
ink it could be mentioned for this release. > What do you think? > > Best, > > On Wed, Sep 21, 2022 at 1:17 AM David Arthur > wrote: > >> Hello Kafka users, developers and client-developers, >> >> This is the second release candidate for Apache Kafka 3.3.0.

  1   2   3   4   5   6   7   8   9   >