Hi, I see that the ticket has been left untouched since a while now. Should it be included in the tiered storage v1? We've observed that lacking a way to throttle uploads to tiered storage has a major impact on producers and consumers when tiered storage access recovers (starving disk IOps/throughput or CPU). For this reason, I think this is an important feature and possibly worth including in v1?
Regards, On Tue, Dec 5, 2023 at 8:43 PM Jun Rao <j...@confluent.io.invalid> wrote: > Hi, Abhijeet, > > Thanks for the KIP. A few comments. > > 10. remote.log.manager.write.quota.default: > 10.1 For other configs, we > use replica.alter.log.dirs.io.max.bytes.per.second. To be consistent, > perhaps this can be sth like remote.log.manager.write.max.bytes.per.second. > 10.2 Could we list the new metrics associated with the new quota. > 10.3 Is this dynamically configurable? If so, could we document the impact > to tools like kafka-configs.sh and AdminClient? > > Jun > > On Tue, Nov 28, 2023 at 2:19 AM Luke Chen <show...@gmail.com> wrote: > > > Hi Abhijeet, > > > > Thanks for the KIP! > > This is an important feature for tiered storage. > > > > Some comments: > > 1. Will we introduce new metrics for this tiered storage quotas? > > This is important because the admin can know the throttling status by > > checking the metrics while the remote write/read are slow, like the rate > of > > uploading/reading byte rate, the throttled time for upload/read... etc. > > > > 2. Could you give some examples for the throttling algorithm in the KIP > to > > explain it? That will make it much clearer. > > > > 3. To solve this problem, we can break down the RLMTask into two smaller > > tasks - one for segment upload and the other for handling expired > segments. > > How do we handle the situation when a segment is still waiting for > > offloading while this segment is expired and eligible to be deleted? > > Maybe it'll be easier to not block the RLMTask when quota exceeded, and > > just check it each time the RLMTask runs? > > > > Thank you. > > Luke > > > > On Wed, Nov 22, 2023 at 6:27 PM Abhijeet Kumar < > abhijeet.cse....@gmail.com > > > > > wrote: > > > > > Hi All, > > > > > > I have created KIP-956 for defining read and write quota for tiered > > > storage. > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-956+Tiered+Storage+Quotas > > > > > > Feedback and suggestions are welcome. > > > > > > Regards, > > > Abhijeet. > > > > > >
