[DISCUSS] Cluster-wide disablement of Tiered Storage

Hello all! I wanted to gather more opinions for https://issues.apache.org/jira/browse/KAFKA-15267 In summary, the problem which I would like to solve is disabling TS (and freeing the resources used by RemoteLog*Manager) because I have decided I no longer want to use it without having to provision

[DISCUSS] KIP-963: Upload and delete lag metrics in Tiered Storage

Hello all! I would like to start a discussion for KIP-963: Upload and delete lag metrics in Tiered Storage (https://cwiki.apache.org/confluence/x/sZGzDw). The purpose of this KIP is to introduce a couple of metrics to track lag with respect to remote storage from the point of view of Kafka. Than

Re: [DISCUSS] KIP-963: Upload and delete lag metrics in Tiered Storage

t; the record size. > > Do you think having the tier-lag by number of segments (or) size of > segments in bytes will be useful > to the operator? > > Thanks, > Kamal > > On Tue, Aug 8, 2023 at 8:56 PM Christo Lolov > wrote: > > > Hello all! > > > > I wo

Re: [VOTE] 3.6.0 RC0

Heya, I have compiled and ran the test target successfully for the 3.6.0-rc0 branch on: Java 11, Scala 2.13 - ARM Java 17, Scala 2.13 - ARM Java 20, Scala 2.18 - ARM Java 11, Scala 2.13 - Intel x86 Java 17, Scala 2.13 - Intel x86 Java 20, Scala 2.13 - Intel x86 I will update the Zookeeper KIP ti

Re: [DISCUSS] KIP-963: Upload and delete lag metrics in Tiered Storage

ose metrics > were not mentioned in the original KIP-405 or KIP-930. Do you think these > would make sense to be added to this KIP and get included in the > discussion? > > Regards, > Gantigmaa > > On Wed, Aug 9, 2023 at 1:53 PM Christo Lolov > wrote: > > > Heya

Re: [DISCUSS] KIP-963: Upload and delete lag metrics in Tiered Storage

metrics and their granularity! Best, Christo On Fri, 13 Oct 2023 at 10:14, Christo Lolov wrote: > Heya Gantigmaa, > > Apologies for the (very) late reply! > > Now that 3.6 has been released and reviewers have a bit more time I will > be picking up this KIP again. I am more

Re: [DISCUSS] KIP-963: Upload and delete lag metrics in Tiered Storage

other internals) to have some metrics attached -- maybe if you > could shared scenarios where this metrics would be useful would be helpful. > > 7. On the metrics naming: not sure the `Total*` prefix is really needed or > adds meaning. When I found it useful is when there are related metr

Re: [DISCUSS] KIP-963: Upload and delete lag metrics in Tiered Storage

I find the record-based lag less useful as records > > > can't > > > > > be relied as a stable unit of measure. So, if we can keep bytes- > and > > > > > segment-based lag, LGTM. > > > > > 3.4. Agree, these metrics should be on the broker

Re: [DISCUSS] KIP-963: Upload and delete lag metrics in Tiered Storage

ake those fine grained remote fetch flow sequence > metrics separately later? > > Thanks, > Satish. > > On Tue, 14 Nov 2023 at 22:07, Christo Lolov > wrote: > > > > Heya everyone, > > > > Apologies for the delay in my response and thank you very much for all

[VOTE] KIP-963: Additional metrics in Tiered Storage

Hello all! Now that the discussion for KIP-963 has winded down, I would like to open it for a vote targeting 3.7.0 as the release. You can find the current version of the KIP at https://cwiki.apache.org/confluence/display/KAFKA/KIP-963%3A+Additional+metrics+in+Tiered+Storage Best, Christo

[DISCUSS] KIP-1005: Add EarliestLocalOffset to GetOffsetShell

Hello all! I would like to start a discussion for https://cwiki.apache.org/confluence/display/KAFKA/KIP-1005%3A+Add+EarliestLocalOffset+to+GetOffsetShell . A new offset called local log start offset was introduced as part of KIP-405: Kafka Tiered Storage. KIP-1005 aims to expose this offset by ch

Re: [DISCUSS] KIP-1005: Add EarliestLocalOffset to GetOffsetShell

> highest-copied-remote-offset via > > GetOffsetShell tool? This will be useful during the debugging session. > > > > > > On Mon, Nov 20, 2023 at 5:38 PM Christo Lolov > > wrote: > > > > > Hello all! > > > > > > I would like to start

Re: [VOTE] KIP-963: Additional metrics in Tiered Storage

Kamal Chandraprakash > > wrote: > > > > > > +1 (non-binding). Thanks for the KIP! > > > > > > On Tue, Nov 21, 2023, 03:04 Divij Vaidya > > wrote: > > > > > > > + 1 (binding) > > > > > > >

Re: [DISCUSS] KIP-1018: Introduce max remote fetch timeout config

Heya! Is it difficult to instead add the metric at kafka.network:type=RequestMetrics,name=TieredStorageMs (or some other name=*)? Alternatively, if it is difficult to add it there, is it possible to add 2 metrics, one at the RequestMetrics level (even if it is total-time-ms - (all other times)) an

Re: [VOTE] KIP-1018: Introduce max remote fetch timeout config

Heya Kamal, Thanks for the KIP and the answers in the discussion! +1 from me :) Best, Christo On Thu, 9 May 2024 at 11:11, Federico Valeri wrote: > +1 non binding > > Thanks > > On Thu, May 9, 2024 at 12:05 PM Luke Chen wrote: > > > > Hi Kamal, > > > > Thanks for the KIP! > > +1 from me. > >

Re: [DISCUSS] KIP-950: Tiered Storage Disablement

ller, not listening for > changes in > >>>> ZK. > >>>> > >>>> 3. In the KRaft handler steps, you said: > >>>> "The controller also updates the Topic metadata to increment the > >>>> tiered_epoch and update the tiere

Re: [DISCUSS] KIP-950: Tiered Storage Disablement

ink infinite retries is a good idea since if a broker is down > forever, this request will never complete. > You mentioned the existing topic deletion is using the similar pattern, how > does it handle this issue? > > Thanks. > Luke > > On Thu, May 9, 2024 at 9:21 PM Christo Lol

[VOTE] KIP-950: Tiered Storage Disablement

Heya! I would like to start a vote on KIP-950: Tiered Storage Disablement in order to catch the last Kafka release targeting Zookeeper - https://cwiki.apache.org/confluence/display/KAFKA/KIP-950%3A++Tiered+Storage+Disablement Best, Christo

Re: [VOTE] KIP-950: Tiered Storage Disablement

Hello! I am closing this vote as ACCEPTED with 3 binding +1 (Luke, Chia-Ping and Satish) and 1 non-binding +1 (Kamal) - thank you for the reviews! Realistically, I don't think I have the bandwidth to get this in 3.8.0. Due to this, I will mark tentatively the Zookeeper part for 3.9 if the communi

Re: [DISCUSS] KIP-950: Tiered Storage Disablement

ood if you could add that into the KIP. > > > > Otherwise, LGTM. > > > > Thank you. > > Luke > > > > On Mon, May 13, 2024 at 11:55 PM Christo Lolov > > wrote: > > > > > Heya! > > > > > > re Kamal - Okay, I believe I un

Re: [VOTE] KIP-950: Tiered Storage Disablement

ture. However, I > am > >> a bit sceptical on the ZK part of the story. The 3.8 release is supposed > >> to > >> be the last one supporting ZK so I don't really see how we could bring > it > >> to ZK, knowing that we don't plan to do a 3.9 release (cur

Re: [VOTE] KIP-1007: Introduce Remote Storage Not Ready Exception

Heya Kamal, The proposed change makes sense to me as it will be a more explicit behaviour than what Kafka does today - I am happy with it! +1 (non-binding) from me Best, Christo On Tue, 12 Dec 2023 at 09:01, Kamal Chandraprakash < kamal.chandraprak...@gmail.com> wrote: > Hi, > > I would like t

[VOTE] KIP-1005: Expose EarliestLocalOffset and TieredOffset

Heya all! KIP-1005 ( https://cwiki.apache.org/confluence/display/KAFKA/KIP-1005%3A+Expose+EarliestLocalOffset+and+TieredOffset) has been open for around a month with no further comments - I would like to start a voting round on it! Best, Christo

Re: [DISCUSS] KIP-1005: Add EarliestLocalOffset to GetOffsetShell

; > -- > Divij Vaidya > > > > On Tue, Nov 21, 2023 at 2:19 PM Christo Lolov > wrote: > > > Heya! > > > > Thanks a lot for this. I have updated the KIP to include exposing the > > tiered-offset as well. Let me know whether the Public Interfaces section > >

Re: [DISCUSS] KIP-1005: Add EarliestLocalOffset to GetOffsetShell

gt; Thank you for making the change Christo. It looks good to me. > > > > -- > > Divij Vaidya > > > > > > > > On Thu, Jan 11, 2024 at 11:19 AM Christo Lolov > > wrote: > > > > > Thank you Divij! > > > > > > I hav

Re: [DISCUSS] KIP-1005: Add EarliestLocalOffset to GetOffsetShell

, if the timestamp greater than recently > committed record timestamp is given.]" > > WDYT? > > Thanks. > Luke > > On Mon, Jan 15, 2024 at 6:46 PM Christo Lolov > wrote: > > > Heya Luke, > > > > Thank for the question! I have expanded in the KIP -

Re: [VOTE] KIP-1005: Expose EarliestLocalOffset and TieredOffset

11 Jan 2024 at 17:52, Divij Vaidya > > > > wrote: > > > > > > > > > > +1 (binding) > > > > > > > > > > Divij Vaidya > > > > > > > > > > > > > > > > > > > > On Tue, Dec 26, 2023

Re: [DISCUSS] KIP-950: Tiered Storage Disablement

Hello! I volunteer to get this KIP moving forward and implemented in Apache Kafka 3.8. I have caught up with Mehari offline and we have agreed that given Apache Kafka 4.0 being around the corner we would like to propose this feature only for KRaft clusters. Any and all reviews and comments are w

Re: [DISCUSS] KIP-950: Tiered Storage Disablement

Duggana > wrote: > > > Hi Christo, > > Thanks for volunteering to contribute to the KIP discussion. I suggest > > considering this KIP for both ZK and KRaft as it will be helpful for > > this feature to be available in 3.8.0 running with ZK clusters. > > > > Thank

Re: [ANNOUNCE] New committer: Christo Lolov

Thank you everyone! It wouldn't have been possible without quite a lot of reviews and extremely helpful inputs from you and the rest of the community! I am looking forward to working more closely with you going forward :) On Tue, 26 Mar 2024 at 14:31, Kirk True wrote: > Congratulations Christo!

Re: [DISCUSS] KIP-1023: Follower fetch from tiered offset

Heya! First of all, thank you very much for the proposal, you have explained the problem you want solved very well - I think a faster bootstrap of an empty replica is definitely an improvement! For my understanding, which concrete offset do you want the leader to give back to a follower - earlies

Re: [DISCUSS] KIP-950: Tiered Storage Disablement

ending Controller -> Broker RPCs. In fact, KIP-631 deprecated these > > RPCs. > > > > > > > > I will come up with some recommendations on how we could improve that > > > one but until then, @Luke please feel free to review the KIP. > > > > > > > > @Satish, if we wan

Re: [DISCUSS] KIP-1023: Follower fetch from tiered offset

> > > Hi Christo, > > > > Thanks for reviewing the KIP. > > > > The follower needs the earliest-pending-upload-offset (and the > > corresponding leader epoch) from the leader. > > This is the first offset the follower will have locally. > > > &

Re: [DISCUSS] KIP-950: Tiered Storage Disablement

tarting doing unexpected upload/fetch tasks, right? > I agree we should make sure in ZK mode, all brokers are completed the > stopPartitions before moving to "Disabled" state because ZK node watcher is > working in a separate thread. But not sure about KRaft mode. > > Thanks

Re: [ANNOUNCE] New Kafka PMC Member: Greg Harris

Congratulations, Greg :) On Mon, 15 Apr 2024 at 07:34, Zhisheng Zhang <31791909...@gmail.com> wrote: > Congratulations Greg! > > > Manikumar 于2024年4月15日周一 13:49写道： > > > Congratulations, Greg. > > > > On Mon, Apr 15, 2024 at 11:18 AM Bruno Cadonna > wrote: > > > > > > Congratulations, Greg! > >

Re: [DISCUSS] KIP-1037: Allow WriteTxnMarkers API with Alter Cluster Permission

Heya Nikhil, Thank you for raising this KIP! Your proposal makes sense to me. In essence you are saying that the permission required by WriteTxnMarkers should be the same as for CreateAcls and DeleteAcls, which is reasonable. If we trust an administrator to assign the correct permissions then we

Re: [DISCUSS] KIP-950: Tiered Storage Disablement

on from DisableRemoteTopic >> within the StopReplicas API which will then disappear in KRaft world as it >> is already scheduled for deprecation. What do you think? >> >> Sounds good to me. >> >> Thanks. >> Luke >> >> On Tue, Apr 9, 2024 at 6:46 PM

Re: [DISCUSS] KIP-950: Tiered Storage Disablement

would have to use already existing methods Let me know if there are any new comments or I have missed something! Best, Christo On Mon, 15 Apr 2024 at 12:40, Christo Lolov wrote: > Heya Doguscan, > > I believe that the state of the world after this KIP will be the following: > >

Re: [VOTE] KIP-1037: Allow WriteTxnMarkers API with Alter Cluster Permission

Heya Nikhil, Thanks for the proposal, as mentioned before it makes sense to me! +1 (binding) Best, Christo On Sat, 20 Apr 2024 at 00:25, Justine Olshan wrote: > Hey Nikhil, > > I meant to comment on the discussion thread, but my draft took so long, you > opened the vote. > > Regardless, I jus

Re: [ANNOUNCE] New committer: Igor Soarez

Congratulations Igor :) ! On Thu, 25 Apr 2024 at 17:07, Igor Soarez wrote: > Thanks everyone, I'm very honoured to join! > > -- > Igor >

Re: [VOTE] KIP-1023: Follower fetch from tiered offset

Heya Abhijeet, Thanks a lot for pushing this forward, especially with the explanation of EARLIEST_PENDING_UPLOAD_OFFSET_TIMESTAMP! +1 from me :) Best, Christo On Fri, 26 Apr 2024 at 12:50, Luke Chen wrote: > Hi Abhijeet, > > Thanks for the KIP. > +1 from me. > > Thanks. > Luke > > On Fri, Apr

Re: [DISCUSS] KIP-1018: Introduce max remote fetch timeout config

Heya Kamal, I quite like the proposal and would support it! However, today I don't think we have a metric which shows the latency of fetch requests which are served from remote, am I wrong? I looked at both https://github.com/clolov/kafka/blob/trunk/core/src/main/scala/kafka/network/RequestChanne

Re: [VOTE] KIP-1075: Introduce delayed remote list offsets purgatory to make LIST_OFFSETS async

Hello! Apologies for the late arrival. I read through the KIP and the discussion thread and I have no concerns with the proposed approach. This is a great improvement! +1 from me. Best, Christo On Fri, 23 Aug 2024 at 02:50, Luke Chen wrote: > +1 from me. > > Thanks. > > On Thu, Aug 22, 2024 a

RE: Last sprint to finish line: Replace EasyMock/Powermock with Mockito

Hello! This is the (roughly) bi-weekly update on the Mockito migration. Firstly, the following PRs have been merged since the last email so thank you to the writers (Yash and Divij) and reviewers (Dalibor, Mickael, Yash, Bruno and Chris): https://github.com/apache/kafka/pull/12459

Re: Last sprint to finish line: Replace EasyMock/Powermock with Mockito

Hello, I have not been able to make a lot of progress on the Mockito migration myself, but Yash and Divij opened and merged a PR each. The following PRs have made it into trunk: https://github.com/apache/kafka/pull/12615 https://github.com/apache/kafka/pull/12677 https:/

RE: Last sprint to finish line: Replace EasyMock/Powermock with Mockito

Hello! This email summarises the current state of Kafka's Mockito migration. The JIRA tickets used to track the progress are https://issues.apache.org/jira/browse/KAFKA-14132 and https://issues.apache.org/jira/browse/KAFKA-14133

Re: [DISCUSS] KIP-895: Dynamically refresh partition count of __consumer_offsets

Greetings, I am bumping the below DISCUSSion thread for KIP-895. The KIP presents a situation where consumer groups are in an undefined state until a rolling restart of a cluster is performed. While I have demonstrated the behaviour using a cluster using Zookeeper I believe the same problem can be

RE: Last sprint to finish line: Replace EasyMock/Powermock with Mockito

Hello! Below you will find the latest state of the Mockito migration. 81% or 39/48 of streams-related tests have been migrated. The last 5 pull requests which are in need of reviews are: * https://github.com/apache/kafka/pull/12449 * https://github.com/apache/kafka/pull/12524 * https://github.com

Re: [DISCUSS] KIP-899: Allow clients to rebootstrap

Hello! Thank you for the KIP. I would like to summarise my understanding of the problem in case I am wrong. Currently a long-running client refreshes their metadata from a set of brokers obtained when first contacting the cluster. If they have been “away” for too long those brokers might have

[DISCUSS] KIP-902: Upgrade Zookeeper to 3.8.1

Hello! I would like to start a discussion for KIP-902: Upgrade Zookeeper to 3.8.1. The Zookeeper version currently used in Kafka reached its end of life in December 2022. You can find the KIP at https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=240882784 Thanks in advance for th

[VOTE] KIP-902: Upgrade Zookeeper to 3.8.1

Hello! I would like to start the vote for KIP-902, which upgrades Zookeeper to version 3.8.1. The KIP can be found at https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=240882784 The discussion thread is https://lists.apache.org/thread/5jbn2x0rtmqz5scyoygbdbj4vo0mpbw1 Thanks Ch

Re: [DISCUSS] KIP-902: Upgrade Zookeeper to 3.8.1

thanks for the great analysis for the ZK upgrading. > > One question: > Since we are going to remove ZK in v4.0, and we don't need the latest > feature in the "current release" ZK 3.8.1, why can't we choose the "stable > release" (3.7.1)? > > Thank y

Re: [DISCUSS] KIP-902: Upgrade Zookeeper to 3.8.1

dered beyond their end of life and > are not maintained. (source: Time Based Release Plan#WhatIsOurEOLPolicy)." > > Ismael > >> On Wed, Feb 15, 2023 at 1:47 AM Christo Lolov >> wrote: >> >> Hello! >> >> I would like to start a discussion for KI

Re: [VOTE] KIP-902: Upgrade Zookeeper to 3.8.1

x is an EOL > > version. > > > > Vote +1 (non binding) > > > > -- > > Divij Vaidya > > > > > > > > On Thu, Feb 23, 2023 at 3:32 PM Christo Lolov > > wrote: > > > >> Hello! > >> > >> I would like to

Re: [DISCUSS] Apache Kafka 3.5.0 release

Hello! What would you suggest as the best way to get more eyes on KIP-902 as I would like it to be included it in 3.5.0? Best, Christo > On 16 Mar 2023, at 10:33, Mickael Maison wrote: > > Hi, > > This is a reminder that KIP freeze is less than a week away (22 Mar). > For a KIP to be consid

Re: [VOTE] KIP-902: Upgrade Zookeeper to 3.8.1

concern better than I could. Best, Christo On Mon, 10 Apr 2023 at 00:53, Colin McCabe wrote: > On Wed, Mar 15, 2023, at 04:58, Christo Lolov wrote: > > Hello Colin, > > > > Thank you for taking the time to review the proposal! > > > > I have attached a compat

Re: [VOTE] KIP-902: Upgrade Zookeeper to 3.8.1

> Overall, I agree with you that we're going to have to move to the new > version of ZK. This fits in with the overall timeline of one more year of > Kafka releases supporting ZK. If Apache Kafka 4.0 is April 2024, we'll need > to be getting security updates for ZK during this t

Re: [DISCUSS] KIP-895: Dynamically refresh partition count of __consumer_offsets

Hello all and thank you for the insightful comments! As pointed out in the third rejected alternative I think the ideal solution is what a few of you have also mentioned - if partitions need to be expanded consumer groups should not lose their offsets and be able to find them. I was aware that

[DISCUSS] KIP-928: Making Kafka resilient to log directories becoming full

Hello all! I would like to start a discussion on KIP-928: Making Kafka resilient to log directories becoming full which can be found at https://cwiki.apache.org/confluence/display/KAFKA/KIP-928%3A+Making+Kafka+resilient+to+log+directories+becoming+full . In summary, I frequently run into problems

Re: [DISCUSS] KIP-905: Broker interceptors

Hello David, Thank you for the proposal - it is an interesting read! I have a few questions about it. 1. Can you take a stance on whether you are proposing the feature just for producers or for consumers as well? If it is just for producers can we remove references to consumers? If it is for bot

RE: [DISCUSS] KIP-858: Handle JBOD broker disk failure in KRaft

Hello Igor! I have been working on a KIP to extend the functionality of JBOD broker disk failures (https://cwiki.apache.org/confluence/display/KAFKA/KIP-928%3A+Making+Kafka+resilient+to+log+directories+becoming+full) and was wondering what is the state of this KIP - were you planning on starti

Re: [DISCUSS] KIP-934: Add DeleteTopicPolicy

Heya Jorge, Thank you for the KIP! This feature sounds great to me since I have encountered problems with this, so I am supporting it. Do you have any idea why the previous KIPs were abandoned - I went through the email conversations and pull requests, but I didn't find a good reason? Best, Chri

Re: [DISCUSS] KIP-935: Extend AlterConfigPolicy with existing configurations

Hello! This proposal will address problems with configuration dependencies which I run into very frequently, so I am fully supporting the development of this feature! Best, Christo On Mon, 22 May 2023 at 17:18, Jorge Esteban Quilcate Otoya < quilcate.jo...@gmail.com> wrote: > Hi everyone, > > I

Re: [VOTE] 3.5.0 RC0

Hey Mickael! I am giving a +1 (non-binding) for this candidate release. * Built from the binary tar.gz source with Java 17 and Scala 2.13 on Intel (m5.4xlarge) and ARM (m6g.4xlarge) machines. * Ran unit and integration tests on Intel and ARM machines. * Ran the Quickstart in both Zookeeper and KR

Re: [DISCUSS] KIP-858: Handle JBOD broker disk failure in KRaft

Heya Igor! I don't have any concerns or suggestions for improvements at this stage - the overall approach makes sense to me! I would be quite interested in attending a call, but as Divij has pointed out the 29th of May is a public holiday, so I won't be able to make that date. If there is another

Re: [DISCUSS] KIP-858: Handle JBOD broker disk failure in KRaft

Heya! 5th of June 16:30 - 17:00 UTC works for me. Best, Christo On Thu, 25 May 2023 at 15:14, Igor Soarez wrote: > Hi Divij, Christo, > > Thank you for pointing that out. > > Let's aim instead for Monday 5th of June, at the same time – 16:30-17:00 > UTC. > > Please let me know if this doesn't

Re: [DISCUSS] KIP-928: Making Kafka resilient to log directories becoming full

Heya Igor, Thank you for reading through the KIP and providing feedback! 11. Good question. I will check whether a change is needed in the processing of the metadata records and come back. My hunch says no as long as the Kafka broker is still alive to process the metadata records. This being said

Re: [DISCUSS] KIP-928: Making Kafka resilient to log directories becoming full

delete a > topic and then bounce the broker that had the disk full, it will delete the > topic directory on startup as part of its snapshot load process. > > So I'm not sure if we really need this. Maybe we should re-evaluate once > we have JBOD + KRaft. > > best, > Colin

Re: [DISCUSS] KIP-928: Making Kafka resilient to log directories becoming full

, Christo Lolov wrote: > Hey Colin, > > Thanks for the review! > > I am also skeptical that much space can be reclaimed via compaction as > detailed in the limitations section of the KIP. > > In my head there are two ways to get out of the saturated state - > configure mo

Re: [VOTE] KIP-937: Improve Message Timestamp Validation

+1 (non-binding) from me as well! This is the type of problem which is difficult to become aware of so the more guardrails we put into place the better. On Wed, 21 Jun 2023 at 23:30, Beyene, Mehari wrote: > Thank you, Justin. That makes sense. > I have updated the KIP to remove the concept of ah

Re: Apache Kafka 3.6.0 release

Hello! I will add KIP-902 to the release plan. I would appreciate a few more reviews on the pull request (https://github.com/apache/kafka/pull/13260) for that KIP as the longer we have it in trunk with tests running against it the more confidence we will have before the release. Best, Christo On

[DISCUSS] KIP-952: Regenerate segment-aligned producer snapshots when upgrading to a Kafka version supporting Tiered Storage

Hello! A customer upgrading from Kafka < 2.8 to the future release 3.6 and wanting to enable tiered storage would have to take responsibility for ensuring that all segments lacking a producer snapshot file have expired and are deleted before enabling the feature. In our experience customers are n

Re: Apache Kafka 3.6.0 release

Hello! A couple of days ago I opened a new KIP for discussion - KIP-952 [1]. I believe it might be a blocker for the release of 3.6.0, but I wanted to bring it up here for a decision on its urgency with the current set of people who are looking at Tiered Storage (Satish, Luke, Ivan, Divij) given t

Re: [DISCUSS] KIP-1058: Txn consumer exerts pressure on remote storage when reading non-txn topic

Heya, Apologies for the delay. I have been thinking about this problem recently as well and while I believe storing a boolean in the metadata is good, I think we can do better by introducing a new method to the RLMM along the lines of Optional nextRemoteLogSegmentMetadataWithTxnIndex(TopicIdParti

Re: [ANNOUNCE] New committer: Kamal Chandraprakash

Many congratulations Kamal! Very well deserved! On Tue, 1 Oct 2024 at 08:21, Jorge Esteban Quilcate Otoya < quilcate.jo...@gmail.com> wrote: > Congrats Kamal!! > > On Tue 1. Oct 2024 at 7.54, Kamal Chandraprakash < > kamal.chandraprak...@gmail.com> wrote: > > > Thank you all for your warm wishes!

Re: New release branch 3.9

Hello, I have filed https://issues.apache.org/jira/browse/KAFKA-17584 as a blocker. While it has not been introduced by a KIP, I think this has durability implications and needs to be addressed immediately. Let me know if you disagree! Best, Christo On Thu, 19 Sept 2024 at 16:01, José Armando G

Re: [VOTE] KIP-1105: Make remote log manager thread-pool configs dynamic

Heya, I have ran a few times into issues which required adjustment of the thread pools and having to carry out a rolling restart was cumbersome. +1 (binding) for the change! Best, Christo On Thu, 7 Nov 2024 at 16:17, Satish Duggana wrote: > +1 (binding) > > On Thu, 7 Nov 2024 at 13:55, Federi

Re: [VOTE] KIP-1058: Txn consumer exerts pressure on remote storage when reading non-txn topic

Heya, Apologies for the delay. You have my +1 vote as well! I am excited to see this making it in 4.0 Best, Christo On Wed, 6 Nov 2024 at 09:43, Kamal Chandraprakash < kamal.chandraprak...@gmail.com> wrote: > Hi all, > > The KIP is passed with 3 (+1) binding votes. Thanks everyone! > > Binding

Re: [ANNOUNCE] New committer: Andrew Schofield

Congratulations Andrew, well deserved! On Mon, 2 Dec 2024 at 16:36, Josep Prat wrote: > Congrats Andrew! > > > On Mon, Dec 2, 2024 at 5:33 PM Mickael Maison > wrote: > > > Congratulations Andrew! > > > > Thanks, > > Mickael > > > > On Mon, Dec 2, 2024 at 5:27 PM Kuan Po Tseng > wrote: > > > >

Re: [DISCUSS] KIP-1126: Serialize changes to Kafka with a build queue

Heya David, Thanks a lot for this! Building on top of Chris's questions - does including only a subset of the stages in the merge queue mean that while subsequent stages (i.e. tests) can run in parallel-ish across PRs they will contain trunk + set of PRs ahead of this one which passed the stage i

Re: [ANNOUNCE] New committer: TengYao Chi

Congratulations! On Thu, 10 Apr 2025 at 18:22, David Arthur wrote: > Congrats!! > > David Arthur > > > On Thu, Apr 10, 2025 at 12:43 TengYao Chi wrote: > > > Thank you very much everyone!! > > > > I truly appreciate all your reviews, guidance, and discussions on my code > > contributions. > > Y

Re: [DISCUSS] KIP-1150 Diskless Topics

Hello! I want to start with saying that this is a big and impressive undertaking and I am really excited to see its progression! I am posting my initial comments in this thread, but they span a few of the child KIPs. Let me know which questions you would like to move elsewhere. I understand that y

Re: [DISCUSS] KIP-1176: Tiered Storage for Active Log Segment

Hello! It is great to see another proposal on the same topic, but optimising for different scenarios, so thanks a lot for the effort put in this! I have a few questions and statements in no particular order. If you use acks=-1 (acks=all) then an acknowledgement can only be sent to the producer i

Re: [VOTE] KIP-1123: Rack-aware partitioning for Kafka Producer

Hello! Apologies for coming late to the party. I think this is a great idea and don't see why we can't try it out in practice! +1 (binding) Best, Christo On Tue, 29 Apr 2025 at 15:16, Andrew Schofield < andrew_schofield_j...@outlook.com> wrote: > Thanks for the KIP, Ivan. > > +1 (binding) > >

Re: 4.0.1 bug-fix release?

Hello! I would like to volunteer as a release manager for 4.0.1! I haven't done this before so I will most likely reach out for help if accepted :) Best, Christo On Wed, 25 Jun 2025 at 16:48, Josep Prat wrote: > But of course, Vivekanand, you are more than welcome to help out testing > and val

[DISCUSS] Apache Kafka 4.0.1 release

Hello! I have volunteered to be the release manager for the 4.0.1 release. I have put together the release plan at https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+4.0.1 The first release candidate will be cut on the 23rd of July (~3 weeks from the date of this email). Feel free to

Re: [DISCUSS] Apache Kafka 4.0.1 release

Hello, The 23rd is now drawing to an end. I am slightly behind schedule, but will be continuing with creating the artifacts tomorrow! Best, Christo On Mon, 30 Jun 2025 at 12:31, Christo Lolov wrote: > Hello! > > I have volunteered to be the release manager for the 4.0.1 release. &

[VOTE] 4.0.1 RC0

Hello Kafka users, developers and client-developers, This is the first candidate for release of Apache Kafka 4.0.1. Since this is a patch version it contains the bug fixes outlined in the release notes below! Release notes for the 4.0.1 release: https://dist.apache.org/repos/dist/dev/kafka/4.0.1

Re: [VOTE] 4.1.0 RC1

Heya! I tested creating a cluster using the binary tgz. Furthermore, I went through the enabling, producing, consuming, and disabling of queues. Lastly, I carried out the same steps on both Docker images. I went through the JavaDocs and clicked on links to confirm they work. I chose artifacts at

[jira] [Created] (KAFKA-15267) Cluster-wide disablement of Tiered Storage

Christo Lolov created KAFKA-15267: - Summary: Cluster-wide disablement of Tiered Storage Key: KAFKA-15267 URL: https://issues.apache.org/jira/browse/KAFKA-15267 Project: Kafka Issue Type: Sub

[jira] [Created] (KAFKA-15298) Disable DeleteRecords on Tiered Storage topics

Christo Lolov created KAFKA-15298: - Summary: Disable DeleteRecords on Tiered Storage topics Key: KAFKA-15298 URL: https://issues.apache.org/jira/browse/KAFKA-15298 Project: Kafka Issue Type

[jira] [Resolved] (KAFKA-15298) Disable DeleteRecords on Tiered Storage topics

[ https://issues.apache.org/jira/browse/KAFKA-15298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Christo Lolov resolved KAFKA-15298. --- Resolution: Won't Fix > Disable DeleteRecords on Tiered Storag

[jira] [Created] (KAFKA-15427) Integration tests in TS test harness detect resource leaks

Christo Lolov created KAFKA-15427: - Summary: Integration tests in TS test harness detect resource leaks Key: KAFKA-15427 URL: https://issues.apache.org/jira/browse/KAFKA-15427 Project: Kafka

[jira] [Resolved] (KAFKA-15399) Enable OffloadAndConsumeFromLeader test

[ https://issues.apache.org/jira/browse/KAFKA-15399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Christo Lolov resolved KAFKA-15399. --- Resolution: Fixed > Enable OffloadAndConsumeFromLeader t

[jira] [Resolved] (KAFKA-15384) Replace EasyMock with Mockito for KTableTransformValuesTest

[ https://issues.apache.org/jira/browse/KAFKA-15384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Christo Lolov resolved KAFKA-15384. --- Resolution: Fixed Closing this ticket in favour of https://issues.apache.org/jira/browse

[jira] [Resolved] (KAFKA-15383) Replace EasyMock with Mockito for KTableImplTest

[ https://issues.apache.org/jira/browse/KAFKA-15383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Christo Lolov resolved KAFKA-15383. --- Resolution: Duplicate Closing this ticket in favour of https://issues.apache.org/jira

[jira] [Resolved] (KAFKA-15382) Replace EasyMock with Mockito for KStreamTransformValuesTest

[ https://issues.apache.org/jira/browse/KAFKA-15382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Christo Lolov resolved KAFKA-15382. --- Resolution: Duplicate Closing this ticket in favour of https://issues.apache.org/jira

[jira] [Resolved] (KAFKA-15385) Replace EasyMock with Mockito for AbstractStreamTest

[ https://issues.apache.org/jira/browse/KAFKA-15385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Christo Lolov resolved KAFKA-15385. --- Resolution: Duplicate Closing this in favour of https://issues.apache.org/jira/browse/KAFKA

[jira] [Created] (KAFKA-15660) File-based Tiered Storage should delete folders upon topic deletion

Christo Lolov created KAFKA-15660: - Summary: File-based Tiered Storage should delete folders upon topic deletion Key: KAFKA-15660 URL: https://issues.apache.org/jira/browse/KAFKA-15660 Project: Kafka

[jira] [Created] (KAFKA-15883) Implement RemoteCopyLagBytes

Christo Lolov created KAFKA-15883: - Summary: Implement RemoteCopyLagBytes Key: KAFKA-15883 URL: https://issues.apache.org/jira/browse/KAFKA-15883 Project: Kafka Issue Type: Sub-task

  1   2   >