[jira] [Created] (KAFKA-19323) RemoteTopicCrudTest to use containsKey for topic config checks

Kamal Chandraprakash created KAFKA-19323: Summary: RemoteTopicCrudTest to use containsKey for topic config checks Key: KAFKA-19323 URL: https://issues.apache.org/jira/browse/KAFKA-19323

Re: [VOTE] KIP-1179: Introduce remote.log.manager.follower.thread.pool.size config

+1 (binding). Thanks for the KIP! On Wed, May 21, 2025 at 9:04 PM Chia-Ping Tsai wrote: > +1 (binding) > > Kuan-Po Tseng 於 2025年5月21日 週三 下午8:10寫道： > > > Hello everyone, > > > > I would like to start a vote on KIP-1179. > > KIP Link: https://cwiki.apache.org/confluence/x/xAlWFQ > > > > Best rega

Re: [ANNOUNCE] Apache Kafka 3.9.1

Goldman Sachs, ING, LinkedIn, Netflix, Pinterest, > Rabobank, > > > Target, The New York Times, Uber, Yelp, and Zalando, among others. > > > > > > A big thank you for the following 57 contributors to this release! > > > (Please report an unintended omission) >

Re: [DISCUSS] KIP-1179: Introduce remote.log.manager.follower.thread.pool.size config

Hi Kuan-Po, The proposal to add the new remote.log.manager.follower.thread.pool.size config, LGTM. Thanks for making this change! -- Kamal On Mon, May 12, 2025 at 10:19 PM KuanPo Tseng wrote: > Hello everyone, > > I’d like to discuss a KIP regarding introducing a new configuration, > remote.lo

Re: [VOTE] KIP-1123: Rack-aware partitioning for Kafka Producer

+1 (binding). Thanks for the KIP! On Mon, May 19, 2025, 16:58 Ivan Yurchenko wrote: > Thank you everyone! > > The KIP is accepted with 5 binding and 3 non-binding votes. > I'll make the implementation PR shortly. > > Best, > Ivan > > > On Thu, May 15, 2025, at 09:33, Luke Chen wrote: > > +1 bind

Re: [DISCUSS] KIP-1178: Introduce remote.max.partition.fetch.bytes in Consumer

y all consumers setting the > same > value. It suggests that they shouldn't be able to provide their own values > at all. > > Thanks, > Andrew > > From: Kamal Chandraprakash > Sent: 08 May 2025 12:07 > To: dev@kafka.apache.org

[DISCUSS] KIP-1178: Introduce remote.max.partition.fetch.bytes in Consumer

Hi all, I've opened the KIP-1178 to add a new config 'remote.max.partition.fetch.bytes' in the consumer. This config allows it to read from remote storage faster. https://cwiki.apache.org/confluence/display/KAFKA/KIP-1178%3A+Introduce+remote.max.partition.fetch.bytes+config+in+Consumer Please ta

[jira] [Reopened] (KAFKA-15777) Configurable remote fetch bytes per partition from Consumer

[ https://issues.apache.org/jira/browse/KAFKA-15777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kamal Chandraprakash reopened KAFKA-15777: -- > Configurable remote fetch bytes per partition from Consu

Re: [DISCUSS] KIP-1105: Make remote log manager thread-pool configs dynamic

RLMFolloweTask, such as > remote.log.manager.follower.thread.pool.size, for consistency? > > 2. Additionally, with the specific config, the size of RLMFolloweTask can > be dynamically configured > > WDYT? > > Best, > Chia-Ping > > > > > Kamal Chandraprakash

Re: [DISCUSS] KIP-1105: Make remote log manager thread-pool configs dynamic

configuration parameter remote.log.manager.thread.pool.size. > > Is there a Jira ticket associated with the deprecation of this setting? > Additionally, is there a replacement configuration available for > followerThreadPool? > > thanks, > Chia-Ping > > On 2024/11/07 16:07:36 Kamal Chan

[jira] [Resolved] (KAFKA-19131) Exception thrown while updating the RemoteLogReader threads

[ https://issues.apache.org/jira/browse/KAFKA-19131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kamal Chandraprakash resolved KAFKA-19131. -- Fix Version/s: 4.1.0 Resolution: Fixed > Exception thrown wh

[jira] [Resolved] (KAFKA-19014) Potential race condition in remote-log-reader and remote-log-index-cleaner thread

[ https://issues.apache.org/jira/browse/KAFKA-19014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kamal Chandraprakash resolved KAFKA-19014. -- Fix Version/s: 4.1.0 Assignee: Kamal Chandraprakash Resolution

[jira] [Resolved] (KAFKA-19180) Fix the hanging testPendingTaskSize

[ https://issues.apache.org/jira/browse/KAFKA-19180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kamal Chandraprakash resolved KAFKA-19180. -- Fix Version/s: 4.1.0 Resolution: Fixed > Fix the hang

[jira] [Resolved] (KAFKA-17184) Remote index cache noisy logging

[ https://issues.apache.org/jira/browse/KAFKA-17184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kamal Chandraprakash resolved KAFKA-17184. -- Fix Version/s: 4.1.0 Resolution: Fixed > Remote index cache no

Re: [ANNOUNCE] New committer: Apoorv Mittal

Congratulations, Apoorv! On Sat, Apr 12, 2025 at 8:23 AM Kuan Po Tseng wrote: > Congrats Apoorv! > > On Sat, Apr 12, 2025 at 9:50 AM Guozhang Wang > wrote: > > > Congratulations Apoorv! > > > > On Fri, Apr 11, 2025 at 3:33 PM Apoorv Mittal > > wrote: > > > > > > Thank you so much everyone! I r

Re: [ANNOUNCE] New committer: TengYao Chi

Congrats, TengYao! Well deserved!! On Sat, Apr 12, 2025 at 7:19 AM Guozhang Wang wrote: > Congratulations TengYao! > > On Fri, Apr 11, 2025 at 4:36 PM Christo Lolov > wrote: > > > > Congratulations! > > > > On Thu, 10 Apr 2025 at 18:22, David Arthur wrote: > > > > > Congrats!! > > > > > > Davi

[jira] [Created] (KAFKA-19131) Exception thrown while updating the RemoteLogReader threads

Kamal Chandraprakash created KAFKA-19131: Summary: Exception thrown while updating the RemoteLogReader threads Key: KAFKA-19131 URL: https://issues.apache.org/jira/browse/KAFKA-19131 Project

Re: [ANNOUNCE] Apache Kafka 4.0.0

> > > >>>> ** Building real-time streaming applications that transform or > > react > > > > >>>> to the streams of data. > > > > >>>> > > > > >>>> Apache Kafka is in use at large and small companies

[jira] [Resolved] (KAFKA-18524) Fix flaky RemoteIndexCacheTest#testCorrectnessForCacheAndIndexFilesWhenResizeCache

[ https://issues.apache.org/jira/browse/KAFKA-18524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kamal Chandraprakash resolved KAFKA-18524. -- Fix Version/s: 4.1.0 Resolution: Duplicate > Fix fl

[jira] [Resolved] (KAFKA-18089) RemoteIndexCacheTest fails with caffeine > 3.1.1

[ https://issues.apache.org/jira/browse/KAFKA-18089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kamal Chandraprakash resolved KAFKA-18089. -- Fix Version/s: 4.1.0 Resolution: Fixed > RemoteIndexCacheTest fa

[jira] [Created] (KAFKA-18825) Fix the flaky EligibleLeaderReplicasIntegrationTest

Kamal Chandraprakash created KAFKA-18825: Summary: Fix the flaky EligibleLeaderReplicasIntegrationTest Key: KAFKA-18825 URL: https://issues.apache.org/jira/browse/KAFKA-18825 Project: Kafka

Re: [VOTE] KIP-1126: Serialize changes to Kafka with a build queue

+1 (binding), Thanks for the KIP! -- Kamal On Fri, Feb 14, 2025 at 10:25 AM Matthias J. Sax wrote: > +1 (binding) > > On 2/13/25 8:03 AM, David Arthur wrote: > > Good point, Chris! The vote will remain open for another 2 days. I will > > resend a vote summary if any additional votes are cast >

[jira] [Created] (KAFKA-18787) RemoteIndexCache fails to remove invalid files on init

Kamal Chandraprakash created KAFKA-18787: Summary: RemoteIndexCache fails to remove invalid files on init Key: KAFKA-18787 URL: https://issues.apache.org/jira/browse/KAFKA-18787 Project: Kafka

[jira] [Resolved] (KAFKA-18728) Move ListOffsetsPartitionStatus to server module

[ https://issues.apache.org/jira/browse/KAFKA-18728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kamal Chandraprakash resolved KAFKA-18728. -- Fix Version/s: 4.1.0 Resolution: Fixed > M

Re: Tagged fields constraint to start from tagId: 0

ly that we will ever have tags as part of the official > protocol reaching that. > > Best, > David > > On Mon, Feb 3, 2025 at 6:36 PM Kamal Chandraprakash < > kamal.chandraprak...@gmail.com> wrote: > > > Bumping up on this thread. > > > > In a forke

Re: Tagged fields constraint to start from tagId: 0

gt; to start with a very large tag-id like 1 for custom tagged fields to not clash with future tagged fields being introduced upstream? TIA. Thanks, Kamal On Thu, Jan 23, 2025 at 11:47 PM Kamal Chandraprakash < kamal.chandraprak...@gmail.com> wrote: > Hi all, > > In KIP-482 > &

[jira] [Created] (KAFKA-18722) Remove the unreferenced methods from TopicBasedRemoteLogMetadataManager

Kamal Chandraprakash created KAFKA-18722: Summary: Remove the unreferenced methods from TopicBasedRemoteLogMetadataManager Key: KAFKA-18722 URL: https://issues.apache.org/jira/browse/KAFKA-18722

Tagged fields constraint to start from tagId: 0

Hi all, In KIP-482 , we have introduced support for optional tagged fields when the API-message supports

Re: [DISCUSS] KIP-1114 Introducing Chunk in Partition

Hi Deo, Thanks for the KIP! "However the limit of messages in a single partition replica is very big. This could lead to very big partitions (~TBs). Moving those partitions are very time consuming and have a big impact on system performance." One way to do faster rebalance is to have a latest-of

Re: [VOTE] KIP-1118: Add Deadlock Protection on Producer Network Thread

+1 (binding). Thanks for the KIP! On Mon, Dec 23, 2024, 08:28 TengYao Chi wrote: > Hi everyone, > > As the vote has been pending for a week, I would like to bump it manually. > Thank you for your attention. > > Sincerely, > TengYao > > Andrew Schofield 於 2024年12月16日 週一 > 下午10:25寫道： > > > +1 (bi

[jira] [Resolved] (KAFKA-17928) Make remote log manager thread-pool configs dynamic

[ https://issues.apache.org/jira/browse/KAFKA-17928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kamal Chandraprakash resolved KAFKA-17928. -- Resolution: Fixed > Make remote log manager thread-pool configs dyna

Re: [ANNOUNCE] New committer: Andrew Schofield

Congrats Andrew! On Mon, Dec 2, 2024 at 7:57 PM Abhinav Dixit wrote: > Congratulations Andrew. Very happy for you :) > > On Mon, Dec 2, 2024 at 7:52 PM TengYao Chi wrote: > > > Congratulations Andrew > > Very well deserved. > > > > > > Mayank Shekhar Narula 於 2024年12月2日 週一 > > 下午10:18寫道： > > >

Re: [VOTE] KIP-1030: Change constraints and default values for various configurations

Hi Divij, +1 (Binding). Thanks for the KIP! Kamal On Thu, Nov 28, 2024, 05:37 Luke Chen wrote: > Hi Divij, > > +1 (binding) from me. > Thanks for the KIP! > > Luke > > On Thu, Nov 28, 2024 at 12:21 AM Divij Vaidya > wrote: > > > Corresponding discussion thread: > > https://lists.apache.org/th

Re: [DISCUSS] KIP-1030: Change constraints and default values for various configurations

pier/expiration tasks, the follower tasks are light-weight so proposing to reduce that thread-pool size to 2. -- Kamal On Wed, Nov 27, 2024 at 11:48 PM Kamal Chandraprakash < kamal.chandraprak...@gmail.com> wrote: > Hi Divij, > > I also share the same concern as Greg pointed out

Re: [DISCUSS] KIP-1030: Change constraints and default values for various configurations

Hi Divij, I also share the same concern as Greg pointed out for segment.bytes / segment.index.bytes. All the tiered storage tests might start to fai

Re: [DISCUSS] KIP-1030: Change constraints and default values for various configurations

ber of cores. > 3. It's now hard for a user to know the actual default value. > 4. It may also be a surprise that the default value changes when a user > adds a listener or a dir. > > Thanks, > > Jun > > On Wed, Nov 20, 2024 at 7:36 AM Kamal Chandraprakash < > ka

Re: [DISCUSS] KIP-1030: Change constraints and default values for various configurations

Hi Divij, We would like to change the default value for the below remote log configurations: 1. remote.log.manager.copier.thread.pool.size -- default value from -1 to 10 2. remote.log.manager.expiration.thread.pool.size -- default value from -1 to 10 In KIP-950

Re: [VOTE] KIP-1106: Add duration based offset reset option for consumer clients

Hi Manikumar, Thanks for the KIP! Duration based offset reset is a useful addition when remote storage is enabled. +1 (binding) Thanks, Kamal On Thu, Nov 14, 2024 at 7:51 PM Apoorv Mittal wrote: > Hi Manikumar, > Thanks for the KIP. It will be a valuable addition. > > +1 (non-binding) > > Reg

Re: [VOTE] KIP-1080: Fix the typo: `maxlifeTimeMs` in CreateDelegationTokenOptions

+1 (binding). Thanks for the KIP! On Mon, Nov 11, 2024 at 9:41 AM Luke Chen wrote: > +1 (binding) from me. > > Thanks for the KIP. > > Luke > > On Mon, Nov 11, 2024 at 10:51 AM TengYao Chi wrote: > > > Hi > > Thanks for KIP. > > +1 (non-binding) > > > > Best, > > TengYao > > > > 黃竣陽 於 2024年11月

Re: [VOTE] KIP-1105: Make remote log manager thread-pool configs dynamic

isto > > > > > > On Thu, 7 Nov 2024 at 16:17, Satish Duggana > > > wrote: > > > > > > > +1 (binding) > > > > > > > > On Thu, 7 Nov 2024 at 13:55, Federico Valeri > > > wrote: > > > > > > > > > &g

[jira] [Created] (KAFKA-17980) Add isReady API to RemoteLogMetadataManager

Kamal Chandraprakash created KAFKA-17980: Summary: Add isReady API to RemoteLogMetadataManager Key: KAFKA-17980 URL: https://issues.apache.org/jira/browse/KAFKA-17980 Project: Kafka

[jira] [Reopened] (KAFKA-17801) RemoteLogManager may compute inaccurate upperBoundOffset for aborted txns

[ https://issues.apache.org/jira/browse/KAFKA-17801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kamal Chandraprakash reopened KAFKA-17801: -- > RemoteLogManager may compute inaccurate upperBoundOffset for aborted t

Re: [VOTE] 3.9.0 RC5

Hi all, I took the 3.9 branch from upstream and ran all the unit tests. The StorageToolTest unit test is failing consistently: % ./gradlew core:test --tests StorageToolTest Gradle Test Run :core:test > Gradle Test Executor 20 > StorageToolTest > testFo

Re: [ANNOUNCE] Apache Kafka 3.9.0

dhwa, Chris Egerton, Christo Lolov, > > Ming-Yen Chung, Colin P. McCabe, Cy, David Arthur, David Jacot, > > demo...@csie.io, dengziming, Dimitar Dimitrov, Dmitry Werner, Dongnuo > Lyu, > > dujian0068, Edoardo Comar, Farbod Ahmadian, Federico Valeri, Fiore Mario > > Vitale, F

Re: [DISCUSS] KIP-1105: Make remote log manager thread-pool configs dynamic

it is already > deprecated, and remote.log.manager.copier.thread.pool.size, > remote.log.manager.expiration.thread.pool.size are derived from that. > > ~Satish. > > > On Thu, 7 Nov 2024 at 10:07, Kamal Chandraprakash > wrote: > > > > Hi all, > > > > If ther

[VOTE] KIP-1105: Make remote log manager thread-pool configs dynamic

Hi all, I'd like to call for a vote on KIP-1105 https://cwiki.apache.org/confluence/display/KAFKA/KIP-1105%3A+Make+remote+log+manager+thread-pool+configs+dynamic Thanks, Kamal

Re: [DISCUSS] KIP-1105: Make remote log manager thread-pool configs dynamic

Hi all, If there are no more comments, then I'll start a voting thread as the change is minor. -- Kamal On Thu, Nov 7, 2024 at 9:00 AM Kamal Chandraprakash < kamal.chandraprak...@gmail.com> wrote: > Hi Federico, > > Updated the KIP by replacing the `isInitialized` to

Re: [DISCUSS] KIP-1105: Make remote log manager thread-pool configs dynamic

4 at 5:22 AM Kamal Chandraprakash > wrote: > > > > Hi Federico, > > > > Thanks for the review! > > > > 1. Changed the API name to `isReady` > > 2. Added an example of stacktrace in the KIP. > > > > PTAL. > > > > Thanks, > > Ka

Re: [DISCUSS] Apache Kafka 4.0.0 release

Hi Luke, Please add the KIP-1058 and KIP-1105 to the release plan. KIP-1058 is accepted and PRs are ready. KIP-1105 is a minor one, and will open a voting thread soon. Thanks, Kamal On Thu, Nov 7, 2024 at 7:28 AM Luke Chen wrote: > Hi Alyssa, > > I've added KIP-966 into the release plan. > > T

Re: [VOTE] KIP-1058: Txn consumer exerts pressure on remote storage when reading non-txn topic

or changes in a backward compatible manner. > > > > ~Satish. > > > > On Mon, 4 Nov 2024 at 19:33, Divij Vaidya > wrote: > > > > > +1 > > > > > > (I have participated in the review and I think we have a good solution > in > > > t

Re: [DISCUSS] KIP-1105: Make remote log manager thread-pool configs dynamic

isReady" could be a better name, instead of > "isInitialized". Wdyt? > > Could you please add an example of the stack trace that the RLMM can > raise during the initialization phase? > > On Sun, Nov 3, 2024 at 4:50 PM Kamal Chandraprakash > wrote: > > >

Re: [DISCUSS] KIP-1058: Txn consumer exerts pressure on remote storage when reading non-txn topic

use the optimization you authored in the PR) > > Thoughts? > > -- > Divij Vaidya > > > > On Mon, Nov 4, 2024 at 3:01 PM Divij Vaidya > wrote: > > > Thanks for patiently addressing all the comments. > > > > I will add the vote in the other thread.

Re: [VOTE] KIP-1058: Txn consumer exerts pressure on remote storage when reading non-txn topic

p 16, 2024 at 7:20 AM Kamal Chandraprakash < > kamal.chandraprak...@gmail.com> wrote: > > > Bumping this thread for vote. PTAL. > > > > On Mon, Sep 9, 2024 at 2:01 PM Kamal Chandraprakash < > > kamal.chandraprak...@gmail.com> wrote: > > > > > H

Re: [DISCUSS] KIP-1058: Txn consumer exerts pressure on remote storage when reading non-txn topic

Hi all, If the review comments are addressed and KIP looks good, please submit your vote on the voting thread. Thanks, Kamal On Sat, Nov 2, 2024 at 7:52 PM Kamal Chandraprakash < kamal.chandraprak...@gmail.com> wrote: > I've opened a PR which adds an integration test > &l

[DISCUSS] KIP-1105: Make remote log manager thread-pool configs dynamic

Hi all, I would like to start a discussion thread on KIP-1105 . This KIP is about 1. Configuring the thread-pool used by the remote-log manager dynamically and 2. Graceful handling o

[jira] [Created] (KAFKA-17928) Make remote log manager thread-pool configs dynamic

Kamal Chandraprakash created KAFKA-17928: Summary: Make remote log manager thread-pool configs dynamic Key: KAFKA-17928 URL: https://issues.apache.org/jira/browse/KAFKA-17928 Project: Kafka

Re: [DISCUSS] KIP-1058: Txn consumer exerts pressure on remote storage when reading non-txn topic

I've opened a PR which adds an integration test <https://github.com/apache/kafka/pull/17668/files#diff-a60b518846fc0f770164d55b6d7cd31e03a002514377578c80c6e22cc120af40R88> to show the impact of this change/KIP. PTAL. -- Kamal On Fri, Nov 1, 2024 at 3:31 PM Kamal Cha

Re: [DISCUSS] KIP-1058: Txn consumer exerts pressure on remote storage when reading non-txn topic

://github.com/apache/kafka/blob/346fdbafc539bc48bb66eedae89a15e240007fd9/core/src/main/java/kafka/log/remote/RemoteLogManager.java#L1801 > . Is there a case where the parameter "offset" will not be equal to the > baseOffset of a segment? > > -- > Divij Vaidya > > > &

Re: [DISCUSS] KIP-1058: Txn consumer exerts pressure on remote storage when reading non-txn topic

we need to assert that we > don't > > read metadata for all segments (i.e. it is not a linear search) from the > > Topic based RLMM. > > 4. in the compatibility section, please document how the existing > clusters > > with Tiered Storage metadata will work d

[jira] [Created] (KAFKA-17830) Cover unit test for TBRLMM init failure cases

Kamal Chandraprakash created KAFKA-17830: Summary: Cover unit test for TBRLMM init failure cases Key: KAFKA-17830 URL: https://issues.apache.org/jira/browse/KAFKA-17830 Project: Kafka

[jira] [Created] (KAFKA-17797) Use DelayedFuturePurgatory for RemoteListOffsetsPurgatory

Kamal Chandraprakash created KAFKA-17797: Summary: Use DelayedFuturePurgatory for RemoteListOffsetsPurgatory Key: KAFKA-17797 URL: https://issues.apache.org/jira/browse/KAFKA-17797 Project

Re: [DISCUSS] KIP-1058: Txn consumer exerts pressure on remote storage when reading non-txn topic

Bump for review. If the additional proposal looks good, I'll append them to the KIP. PTAL. New API in RLMM#nextRemoteLogSegmentMetadataWithTxnIndex -- Kamal On Sun, Oct 6, 2024 at 7:20 PM Kamal Chandraprakash < kamal.chandraprak...@gmail.com> wrote: > Hi Christo, > > T

Re: [DISCUSS] KIP-1058: Txn consumer exerts pressure on remote storage when reading non-txn topic

gt; Best, > Christo > > On Fri, 4 Oct 2024 at 10:48, Kamal Chandraprakash < > kamal.chandraprak...@gmail.com> wrote: > > > Hi Luke, > > > > Thanks for the review! > > > > > Do you think it is helpful if we store the "least abort start offset

Re: [DISCUSS] KIP-1058: Txn consumer exerts pressure on remote storage when reading non-txn topic

bool value in the segment > metadata. > Do you think it is helpful if we store the "least abort start offset in the > segment", and -1 means no txnIndex. So that we can have a way to know if we > need to fetch this txn index or not. > > Thanks. > Luke > > On M

Re: [ANNOUNCE] New committer: Kamal Chandraprakash

tulations Kamal! > > > > > > On Mon, 30 Sept, 2024, 18:07 Luke Chen, wrote: > > > > > >> Hi all, > > >> > > >> The PMC of Apache Kafka is pleased to announce a new Kafka committer, > > Kamal > > >> Chandraprakash.

[jira] [Created] (KAFKA-17637) Invert the search for LIST_OFFSETS request for remote storage topic

Kamal Chandraprakash created KAFKA-17637: Summary: Invert the search for LIST_OFFSETS request for remote storage topic Key: KAFKA-17637 URL: https://issues.apache.org/jira/browse/KAFKA-17637

[jira] [Resolved] (KAFKA-15266) Static configs set for non primary synonyms are ignored for Log configs

[ https://issues.apache.org/jira/browse/KAFKA-15266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kamal Chandraprakash resolved KAFKA-15266. -- Resolution: Duplicate > Static configs set for non primary synonyms

Re: [DISCUSS] KIP-1021: Allow to get last stable offset (LSO) in kafka-get-offsets.sh

hen wrote: > Hi Ahmed, > > Are you still interested in this KIP? > Should we take it over if you're busy on other things? > > Thanks. > Luke > > > On Wed, Mar 27, 2024 at 12:26 PM Kamal Chandraprakash < > kamal.chandraprak...@gmail.com> wrote: > >

Re: [VOTE] KIP-1058: Txn consumer exerts pressure on remote storage when reading non-txn topic

Bumping this thread for vote. PTAL. On Mon, Sep 9, 2024 at 2:01 PM Kamal Chandraprakash < kamal.chandraprak...@gmail.com> wrote: > Hi all, > > I'd like to open voting for KIP-1058. This KIP improves the consumer > reading from remote storage when READ_COMMITTED isolation l

[jira] [Created] (KAFKA-17552) Handle LIST_OFFSETS request for max_timestamp when remote storage is enabled

Kamal Chandraprakash created KAFKA-17552: Summary: Handle LIST_OFFSETS request for max_timestamp when remote storage is enabled Key: KAFKA-17552 URL: https://issues.apache.org/jira/browse/KAFKA-17552

Re: [DISCUSS] Single broker failures causing offline partitions

Hi Haruki, We are also interested in this issue. > The problem is how to identify such "eligible" replicas. Is the leader election automated to find the replica with the highest offset and latest epoch? If yes, could you please open a PR for it? When a broker goes down, it might be serving lead

[VOTE] KIP-1058: Txn consumer exerts pressure on remote storage when reading non-txn topic

Hi all, I'd like to open voting for KIP-1058. This KIP improves the consumer reading from remote storage when READ_COMMITTED isolation level is enabled. PTAL. KIP-1058

Re: [ANNOUNCE] New committer: Jeff Kim

Congrats, Jeff! On Mon, Sep 9, 2024 at 12:20 PM Chia-Ping Tsai wrote: > Congratulations, Jeff! thanks for you to bring the great new group > coordinator! > > Best, > Chia-Ping > > David Jacot 於 2024年9月9日 週一 下午2:44寫道： > > > Hi all, > > > > The PMC of Apache Kafka is pleased to announce a new Ka

Re: [VOTE] KIP-1075: Introduce delayed remote list offsets purgatory to make LIST_OFFSETS async

Hi all, The vote is closed. The KIP has been accepted with 4 +1 binding votes (Satish, Luke, Christo, and Chia-Ping). Thank you all for the participation! -- Kamal On Thu, Aug 29, 2024 at 5:14 PM Chia-Ping Tsai wrote: > +1 (binding) > > On 2024/08/19 05:09:27 Kamal Chandraprak

Re: [DISCUSS] KIP-1075: Introduce delayed remote list offsets purgatory to make LIST_OFFSETS async

Hi Chia-Ping, Thanks for the review! This is a good point to add provision to convey the timeout from client to the server. Updated the KIP to add a new 'TimeoutMs' field in the ListOffsetsRequest and bumped the version to 10. To retain the existing ListOffsets request behaviour: 1. AdminClient

Re: [ANNOUNCE] New committer: Lianet Magrans

Congrats Lianet! On Thu, Aug 29, 2024 at 8:07 AM Luke Chen wrote: > Congratulations Lianet! > > Luke > > On Thu, Aug 29, 2024 at 9:54 AM Lianet M. wrote: > > > Thank you very much everyone! It truly takes the great shared knowledge > you > > all put out there with amazing reviews and discussion

Re: [DISCUSS] KIP-1075: Introduce delayed remote list offsets purgatory to make LIST_OFFSETS async

Hi all, This KIP needs one more binding vote to pass. Please review the KIP. Thanks! -- Kamal On Sat, Aug 24, 2024 at 8:11 AM Kamal Chandraprakash < kamal.chandraprak...@gmail.com> wrote: > Hi Jun, > > Thanks for the review! > > 1. purgatoryName = RemoteListOffsets >

Re: [DISCUSS] KIP-1075: Introduce delayed remote list offsets purgatory to make LIST_OFFSETS async

al level like the fetch > request? > > Jun > > On Fri, Aug 2, 2024 at 12:04 AM Kamal Chandraprakash < > kamal.chandraprak...@gmail.com> wrote: > > > Hi all, > > > > I would like to start a discussion thread on KIP-1075 > > < > >

[VOTE] KIP-1075: Introduce delayed remote list offsets purgatory to make LIST_OFFSETS async

Hi, I would like to open voting for KIP-1075. I've addressed the review comments in the discussion thread. Please vote if the proposal looks good. https://cwiki.apache.org/confluence/plugins/servlet/mobile?contentId=315494171#content/view/315494171 Thanks, Kamal

Re: [DISCUSS] KIP-1075: Introduce delayed remote list offsets purgatory to make LIST_OFFSETS async

Hi all, If there are no more comments, I'll start a vote soon. Thanks, Kamal On Wed, Aug 14, 2024 at 5:05 PM Luke Chen wrote: > Hi Kamal, > > Thanks for the update. > LGTM. > > Luke > > On Wed, Aug 14, 2024 at 7:25 PM Kamal Chandraprakash < > kamal.chandr

Re: [DISCUSS] KIP-1075: Introduce delayed remote list offsets purgatory to make LIST_OFFSETS async

Hi all, > I saw we added some new configs/metrics. I have removed the recent changes to the public interfaces to limit the scope of the KIP to minimum. PTAL. Thanks, Kamal On Wed, Aug 14, 2024 at 9:58 AM Kamal Chandraprakash < kamal.chandraprak...@gmail.com> wrote: > Hi Luk

Re: [DISCUSS] KIP-1075: Introduce delayed remote list offsets purgatory to make LIST_OFFSETS async

we be consistent to use > `reader`, instead of `read`? > > LC7: We should set a default value for the newly introduced configs and > written in KIP. > > Thanks. > Luke > > On Tue, Aug 13, 2024 at 8:47 PM Kamal Chandraprakash < > kamal.chandraprak...@gmail.com> wro

Re: [DISCUSS] KIP-1058: Txn consumer exerts pressure on remote storage when reading non-txn topic

Bumping this thread for KIP review! We can go for the simplest solution that is proposed in this KIP and it can be improved in the subsequent iteration. PTAL. Thanks, Kamal On Fri, Aug 2, 2024 at 11:42 AM Kamal Chandraprakash < kamal.chandraprak...@gmail.com> wrote: > Hi Divij, >

Re: [DISCUSS] KIP-1075: Introduce delayed remote list offsets purgatory to make LIST_OFFSETS async

receives the "response" from > server? > > LC4: I found the different consumer and admin behavior when setting > "request.timeout" and "default.api.timeout" is confusing. Are they expected > or a bug? > > Thank you. > Luke > > > On Thu, Aug 8

Re: [DISCUSS] KIP-1075: Introduce delayed remote list offsets purgatory to make LIST_OFFSETS async

Hi Luke, Thanks for the review! LC1: When the consumer starts to read data, then it might need the below offsets: earliest, latest, and last-committed-offset based on the "auto.offset.reset" config. The earliest and latest offsets have special timestamps -2 and -1, those timestamp corresponding

Re: [VOTE] KIP-950: Tiered Storage Disablement

which > > means `retention.ms` value to be used) > > > > So, basically, we don't change the original design, just want to make > sure > > users are aware of the retention policy change after disabling remote log > > copy. > > > > Let me know if you have

Re: [DISCUSS] KIP-1075: Introduce delayed remote list offsets purgatory to make LIST_OFFSETS async

Bumping this thread. Please take a look. On Fri, Aug 2, 2024 at 12:32 PM Kamal Chandraprakash < kamal.chandraprak...@gmail.com> wrote: > Hi all, > > I would like to start a discussion thread on KIP-1075 > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-1075%3A+Int

[DISCUSS] KIP-1075: Introduce delayed remote list offsets purgatory to make LIST_OFFSETS async

Hi all, I would like to start a discussion thread on KIP-1075 to make the remote LIST_OFFSETS an async operation. The KIP is here: https://cwiki.apache.org/co

Re: [DISCUSS] KIP-1058: Txn consumer exerts pressure on remote storage when reading non-txn topic

we would not have to > iterate through the metadata for the dominant case where the partition had > no transactions at all. > > -- > Divij Vaidya > > > > On Tue, Jun 25, 2024 at 11:42 AM Kamal Chandraprakash < > kamal.chandraprak...@gmail.com> wrote: > &g

Re: [kafka-clients] [ANNOUNCE] Apache Kafka 3.8.0

wxy, Dung Ha, Edoardo Comar, Eduwer > >> Camacaro, Emanuele Sabellico, Erik van Oosten, Eugene Mitskevich, Fan > >> Yang, Federico Valeri, Fiore Mario Vitale, flashmouse, Florin Akermann, > >> Frederik Rouleau, Gantigmaa Selenge, Gaurav Narula, ghostspiders, > >>

Re: [VOTE] KIP-950: Tiered Storage Disablement

et `remote.copy.disabled= > true`. If you want to disable remote storage and delete all remote data, > please set `remote.storage.enable=false,remote.log.delete.on.disable=true`. > > I've updated the KIP. Please take a look when available. > > https://cwiki.apache.org/confl

Re: [VOTE] KIP-950: Tiered Storage Disablement

Correction: (2): Wait for all the remote segments to be deleted async due to breach by retention time (or) size, then set the `remote.storage.enable = false` and `remote.log.delete.on.disable = true`. This step is optional. On Thu, Jul 25, 2024 at 11:13 PM Kamal Chandraprakash

Re: [VOTE] KIP-950: Tiered Storage Disablement

Hi Chia-Ping, Thanks for the review! > If so, what is the purpose of `remote.log.delete.on.disable=false`? IIUC, the purpose of `remote.log.delete.on.disable` is to get explicit confirmation from the user before deleting the remote log segments. The concern raised in the thread is that if the u

Re: [DISCUSS] KIP-1051 Statically configured log replication throttling

about having it as a gauge for number of partitions that are > currently leader of follower throttled (similar to the URP metric)? > > Kind regards, > Harry > > On Thursday, 11 July 2024 at 19:02, Kamal Chandraprakash < > kamal.chandraprak...@gmail.co

Re: [DISCUSS] KIP-1066: Mechanism to cordon brokers and log directories

rectories cordoned, and > > you still need to describe a specific broker's configuration to find > > the "name" of a log directory you want to cordon. > > > > I think an easy way to get an overall view of the cordoned log > > directories/brokers will

Re: [DISCUSS] KIP-1066: Mechanism to cordon brokers and log directories

Hi Mickael, Thanks for the KIP! This is a useful feature which helps to decommission the nodes by essentially creating a new replica exclude broker list. To cordon a list of brokers, we have to apply the config on each of the broker nodes and similarly to see the list of cordoned brokers, we hav

Re: [DISCUSS] KIP-1051 Statically configured log replication throttling

Hi Harry Fallows, Thanks for the KIP! I went over both the KIP-1051 and KIP-1009. Assuming that the leader.replication.throttled.replicas and follower.replication.throttled.replicas are set to Wildcard (*) to apply for all the partitions in the broker. If we set a static value for leader and foll

Re: [DISCUSS] KIP-1058: Txn consumer exerts pressure on remote storage when reading non-txn topic

Bump. Please review this proposal. On Mon, Jun 17, 2024 at 6:55 PM Kamal Chandraprakash < kamal.chandraprak...@gmail.com> wrote: > Divij, > > Thanks for the review! Updated the KIP with 1, 2, 3, and 4 review > comments. > > > 4. Potential alternative - Instead of h

Re: [VOTE] KIP-1057: Add remote log metadata flag to the dump log tool

Hi Federico, Thanks for the KIP! +1 from me. On Fri, Jun 21, 2024 at 5:47 PM Luke Chen wrote: > Hi Fede, > > Thanks for the KIP! > +1 from me. > > Luke > > On Fri, Jun 21, 2024 at 6:44 PM Federico Valeri > wrote: > > > Hi all, I'd like to kick off a vote on KIP-1057. > > > > Design doc: > > >

Re: [DISCUSS] KIP-1057: Add remote log metadata flag to the dump log tool

e dump-logs tool with custom decoder > > - Running kafka-consumer.sh on the topic. > > > > ~Satish. > > > > > > ~Satish. > > > > > > > > On Mon, 17 Jun 2024 at 15:55, Federico Valeri > > wrote: > > > > > > Hi Kamal,

  1   2   3   >