[VOTE] KIP-930: Tiered Storage Metrics

Hi All, I would like to start the vote for KIP-930 Tiered Storage Metrics. The KIP is here: https://cwiki.apache.org/confluence/display/KAFKA/KIP-930%3A+Tiered+Storage+Metrics Regards Abhijeet.

Re: [DISCUSS] KIP-930: Tiered Storage Metrics

t-offset and log-end-offset metrics. Should we > expose local-log-start-offset and highest-offset-uploaded-to-remote-storage > as metric? > > Thanks, > Kamal > > On Mon, Jul 24, 2023 at 2:08 PM Abhijeet Kumar > > wrote: > > > Hi All, > > > > I created

Re: [VOTE] KIP-930: Tiered Storage Metrics

Please find the updated link to the KIP: https://cwiki.apache.org/confluence/display/KAFKA/KIP-930%3A+Rename+ambiguous+Tiered+Storage+Metrics Updated the KIP as per our conversation on the discussion thread. On Tue, Jul 25, 2023 at 11:29 AM Abhijeet Kumar wrote: > Hi All, > > I woul

Re: [VOTE] KIP-930: Tiered Storage Metrics

to it yet. > > > > -- > > Divij Vaidya > > > > > > > > On Tue, Jul 25, 2023 at 6:45 PM Satish Duggana > wrote: > > > > > > +1 for the KIP. > > > > > > Thanks, > > > Satish. > > > > > > On Tue

[DISCUSS] KIP-956: Tiered Storage Quotas

Hi All, I have created KIP-956 for defining read and write quota for tiered storage. https://cwiki.apache.org/confluence/display/KAFKA/KIP-956+Tiered+Storage+Quotas Feedback and suggestions are welcome. Regards, Abhijeet.

Re: [VOTE] KIP-1023: Follower fetch from tiered offset

gt; On Fri, 26 Apr 2024 at 8:35 PM, Jun Rao wrote: > > > Hi, Abhijeet, > > > > Thanks for the KIP. +1 > > > > Jun > > > > On Thu, Apr 25, 2024 at 10:30 PM Abhijeet Kumar < > > abhijeet.cse@gmail.com> > > wrote: > > > > >

Re: [DISCUSS] KIP-1023: Follower fetch from tiered offset

and now the new > last-tiered-offset is 15 > 5. Go back to 2 > > This could cause a cycle where the replica will be stuck trying to > reconcile with the leader. > > -- > Divij Vaidya > > > > On Fri, Apr 26, 2024 at 7:28 AM Abhijeet Kumar > > wrote: > >

Re: [DISCUSS] KIP-1023: Follower fetch from tiered offset

gt; documented as an acceptable risk in current design. > > -- > Divij Vaidya > > > > On Tue, Jul 23, 2024 at 11:55 AM Abhijeet Kumar < > abhijeet.cse@gmail.com> > wrote: > > > Hi Divij, > > > > Seems like there is some confusion about the ne

Re: [DISCUSS] KIP-1023: Follower fetch from tiered offset

r this scenario due to increased probability of having > an aggressive local cleanup with Tiered Storage. > > -- > Divij Vaidya > > > > On Wed, Jul 24, 2024 at 1:22 PM Abhijeet Kumar > > wrote: > > > Hi Divij, > > > > The rare scenario we are discu

Re: [DISCUSS] KIP-956: Tiered Storage Quotas

the situation when a segment is still waiting for > > > offloading while this segment is expired and eligible to be deleted? > > > Maybe it'll be easier to not block the RLMTask when quota exceeded, and > > > just check it each time the RLMTask runs? > > >

Re: [DISCUSS] KIP-956: Tiered Storage Quotas

RLMTask into smaller tasks is already being proposed in KIP-950. The two tasks will need some coordination at a topic-partition level to handle such cases as pointed out. For this particular case, the upload task could skip uploading the segment if it is already eligible for deletion. > Thank

Re: [DISCUSS] KIP-956: Tiered Storage Quotas

andle the situation when a segment is still waiting for > > offloading while this segment is expired and eligible to be deleted? > > Maybe it'll be easier to not block the RLMTask when quota exceeded, and > > just check it each time the RLMTask runs? > > >

Re: [DISCUSS] KIP-956: Tiered Storage Quotas

tive Kafka topics.". However, this > can happen with or without tier storage and the current quota system for > consumers is designed for solving this exact problem. Could you explain the > usage of this additional quota? > > Thanks, > > Jun > > On Mon, Feb 12, 202

Re: [DISCUSS] KIP-956: Tiered Storage Quotas

ger, name=RemoteFetchThrottleTime > kafka.log.remote:type=RemoteLogManager, name=RemoteCopyThrottleTime > > Jun > > On Tue, Feb 27, 2024 at 1:39 AM Abhijeet Kumar > > wrote: > > > Hi Jun, > > > > The existing quota system for consumers is designed

Re: [DISCUSS] KIP-956: Tiered Storage Quotas

s use read/write whereas the metrics use fetch/copy. Could > we make them consistent? > > Jun > > On Wed, Feb 28, 2024 at 6:49 AM Abhijeet Kumar > > wrote: > > > Hi Jun, > > > > Clarified the meaning of the two metrics. Also updated the KIP. > > > >

Re: [DISCUSS] KIP-956: Tiered Storage Quotas

look, and it LGTM now! > > Thanks. > Luke > > On Tue, Mar 5, 2024 at 2:50 AM Jun Rao wrote: > > > Hi, Abhijeet, > > > > Thanks for the reply. Sounds good to me. > > > > Jun > > > > > > On Sat, Mar 2, 2024 at 7:40 PM Abhijeet Kumar <

[VOTE] KIP-956: Tiered Storage Quotas

Hi All, I would like to start the vote for KIP-956 - Tiered Storage Quotas The KIP is here: https://cwiki.apache.org/confluence/display/KAFKA/KIP-956+Tiered+Storage+Quotas Regards. Abhijeet.

Re: [DISCUSS] KIP-956: Tiered Storage Quotas

the configs: > - RLMCopyQuotaManager > - RLMFetchQuotaManager > > > On Fri, 8 Mar 2024 at 13:43, Abhijeet Kumar > wrote: > > > Thank you all for your comments. As all the comments in the thread are > > addressed, I am starting a Vote thread for the KIP. Please have a look. >

Re: [VOTE] KIP-956: Tiered Storage Quotas

t; > > > > -- > > > > Kamal > > > > > > > > On Fri, Mar 8, 2024 at 11:02 PM Jun Rao > > > wrote: > > > > > > > > > Hi, Abhijeet, > > > > > > > > > > Thanks for the KIP.

[DISCUSS] KIP-1023: Follower fetch from tiered offset

Hi All, I have created KIP-1023 to introduce follower fetch from tiered offset. This feature will be helpful in significantly reducing Kafka rebalance/rebuild times when the cluster is enabled with tiered storage. https://cwiki.apache.org/confluence/display/KAFKA/KIP-1023%3A+Follower+fetch+from+t

Re: [DISCUSS] KIP-1023: Follower fetch from tiered offset

> back to a follower - earliest-pending-upload-offset or the > latest-tiered-offset? If it is the second, then I believe KIP-1005 ought to > already be exposing that offset as part of the ListOffsets API, no? > > Best, > Christo > > On Wed, 27 Mar 2024 at 18:23, Abhijeet Kumar >

Re: [DISCUSS] KIP-1023: Follower fetch from tiered offset

e follower know Local-Log-Start-Offset? > Missed this detail. The follower will need to call the leader APi to fetch the EarliestLocal offset for this. > Jun > > On Sat, Mar 30, 2024 at 5:51 AM Abhijeet Kumar > > wrote: > > > Hi Christo, > > > >

Re: [DISCUSS] KIP-1023: Follower fetch from tiered offset

the offset and the corresponding epoch. > > > > 4. "Check if the follower replica is empty and if the feature to use > > last-tiered-offset is enabled." > > Why do we need to check if the follower replica is empty? > > > > 5. It can be confirmed by ch

Re: [DISCUSS] KIP-1023: Follower fetch from tiered offset

we add "Rejected Alternatives" section to the end of the KIP to > add some of them? > Like the "ListOffsetRequest" approach VS "Earliest-Pending-Upload-Offset" > approach, or getting the "Earliest-Pending-Upload-Offset" from remote log > metadata

Re: [DISCUSS] KIP-1023: Follower fetch from tiered offset

follower? > > > > > > 3. Could we add "Rejected Alternatives" section to the end of the KIP > to > > > add some of them? > > > Like the "ListOffsetRequest" approach VS > "Earliest-Pending-Upload-Offset" > > > approach,

Re: [DISCUSS] KIP-1023: Follower fetch from tiered offset

ments. > > Luke > > On Thu, Apr 25, 2024 at 4:21 AM Jun Rao wrote: > > > Hi, Abhijeet, > > > > Thanks for the updated KIP. It looks good to me. > > > > Jun > > > > On Mon, Apr 22, 2024 at 12:08 PM Abhijeet Kumar < > > abhijeet.cs

[VOTE] KIP-1023: Follower fetch from tiered offset

Hi All, I would like to start the vote for KIP-1023 - Follower fetch from tiered offset The KIP is here: https://cwiki.apache.org/confluence/display/KAFKA/KIP-1023%3A+Follower+fetch+from+tiered+offset Regards. Abhijeet.

[DISCUSS] KIP-930: Tiered Storage Metrics

Hi All, I created KIP-930 for adding RemoteIndexCache stats and also to rename some tiered storage metrics added as part of KIP-405 to remove ambiguity. https://cwiki.apache.or

[DISCUSS] KIP-930: Tiered Storage Metrics

Hi All, I created KIP-930 for adding RemoteIndexCache stats and also to rename some tiered storage metrics added as part of KIP-405 to remove ambiguity. https://cwiki.apache.or

[jira] [Created] (KAFKA-15245) Improve Tiered Storage Metrics

Abhijeet Kumar created KAFKA-15245: -- Summary: Improve Tiered Storage Metrics Key: KAFKA-15245 URL: https://issues.apache.org/jira/browse/KAFKA-15245 Project: Kafka Issue Type: Sub-task

[jira] [Created] (KAFKA-15260) RLM Task should wait until RLMM is initialized before copying segments to remote

Abhijeet Kumar created KAFKA-15260: -- Summary: RLM Task should wait until RLMM is initialized before copying segments to remote Key: KAFKA-15260 URL: https://issues.apache.org/jira/browse/KAFKA-15260

[jira] [Created] (KAFKA-15261) ReplicaFetcher thread should not block if RLMM is not initialized

Abhijeet Kumar created KAFKA-15261: -- Summary: ReplicaFetcher thread should not block if RLMM is not initialized Key: KAFKA-15261 URL: https://issues.apache.org/jira/browse/KAFKA-15261 Project: Kafka

[jira] [Created] (KAFKA-15293) Update metrics doc to add tiered storage metrics

Abhijeet Kumar created KAFKA-15293: -- Summary: Update metrics doc to add tiered storage metrics Key: KAFKA-15293 URL: https://issues.apache.org/jira/browse/KAFKA-15293 Project: Kafka Issue

[jira] [Created] (KAFKA-15405) Create a new error code to indicate a resource is not ready yet

Abhijeet Kumar created KAFKA-15405: -- Summary: Create a new error code to indicate a resource is not ready yet Key: KAFKA-15405 URL: https://issues.apache.org/jira/browse/KAFKA-15405 Project: Kafka

[jira] [Resolved] (KAFKA-15261) ReplicaFetcher thread should not block if RLMM is not initialized

[ https://issues.apache.org/jira/browse/KAFKA-15261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhijeet Kumar resolved KAFKA-15261. Resolution: Fixed > ReplicaFetcher thread should not block if RLMM is not initiali

[jira] [Resolved] (KAFKA-15181) Race condition on partition assigned to TopicBasedRemoteLogMetadataManager

[ https://issues.apache.org/jira/browse/KAFKA-15181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhijeet Kumar resolved KAFKA-15181. Resolution: Fixed > Race condition on partition assigned

[jira] [Created] (KAFKA-16706) Refactor ReplicationQuotaManager/RLMQuotaManager to eliminate code duplication

Abhijeet Kumar created KAFKA-16706: -- Summary: Refactor ReplicationQuotaManager/RLMQuotaManager to eliminate code duplication Key: KAFKA-16706 URL: https://issues.apache.org/jira/browse/KAFKA-16706

[jira] [Created] (KAFKA-17069) Remote copy throttle metrics

Abhijeet Kumar created KAFKA-17069: -- Summary: Remote copy throttle metrics Key: KAFKA-17069 URL: https://issues.apache.org/jira/browse/KAFKA-17069 Project: Kafka Issue Type: Sub-task

[jira] [Created] (KAFKA-17108) ListOffset API changes

Abhijeet Kumar created KAFKA-17108: -- Summary: ListOffset API changes Key: KAFKA-17108 URL: https://issues.apache.org/jira/browse/KAFKA-17108 Project: Kafka Issue Type: Sub-task

[jira] [Resolved] (KAFKA-16853) Split RemoteLogManagerScheduledThreadPool

[ https://issues.apache.org/jira/browse/KAFKA-16853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhijeet Kumar resolved KAFKA-16853. Resolution: Fixed > Split RemoteLogManagerScheduledThreadP

[jira] [Created] (KAFKA-17266) Add dynamic broker config to enabled tiered offset follower fetch

Abhijeet Kumar created KAFKA-17266: -- Summary: Add dynamic broker config to enabled tiered offset follower fetch Key: KAFKA-17266 URL: https://issues.apache.org/jira/browse/KAFKA-17266 Project: Kafka

[jira] [Created] (KAFKA-17302) ReplicaFetcher changes for fetching from tiered offset

Abhijeet Kumar created KAFKA-17302: -- Summary: ReplicaFetcher changes for fetching from tiered offset Key: KAFKA-17302 URL: https://issues.apache.org/jira/browse/KAFKA-17302 Project: Kafka

[jira] [Created] (KAFKA-17313) Fix object metric name for Tiered Storage quotas

Abhijeet Kumar created KAFKA-17313: -- Summary: Fix object metric name for Tiered Storage quotas Key: KAFKA-17313 URL: https://issues.apache.org/jira/browse/KAFKA-17313 Project: Kafka Issue

[jira] [Created] (KAFKA-17344) Add Unit tests for empty follower fetch scenarios

Abhijeet Kumar created KAFKA-17344: -- Summary: Add Unit tests for empty follower fetch scenarios Key: KAFKA-17344 URL: https://issues.apache.org/jira/browse/KAFKA-17344 Project: Kafka Issue

[jira] [Created] (KAFKA-15236) Rename Remote Storage metrics to remove ambiguity

Abhijeet Kumar created KAFKA-15236: -- Summary: Rename Remote Storage metrics to remove ambiguity Key: KAFKA-15236 URL: https://issues.apache.org/jira/browse/KAFKA-15236 Project: Kafka Issue