Hi Guang, Thanks for the question. You hit on exactly why we need to view these as a set.
Short answer is yes, Diskless absolutely needs these metrics, primarily to track cache efficiency. In the Diskless design (KIP-1163), the local broker disk is essentially an LRU cache for S3. If a consumer reads data that isn't locally cached, it triggers a fetch from object storage. That is the expensive part. KIP-1267 read metrics like RemoteFetchBytes effectively become our cache miss billing meter. Without them, we would see high S3 GET costs but have no way to attribute them to specific topics or tenants causing the cache thrashing. You are right that the current KIP focuses heavily on the read path. For Diskless, the write path is also 100% remote, so we probably need to ensure RemoteUploadBytes is also broken down per-topic to capture the full ingest cost. Since both architectures likely share the RemoteStorageManager interface, hooking these metrics in there future-proofs us for both KIP-405 and KIP-1150. Regards, Viquar Khan https://www.linkedin.com/in/vaquar-khan-b695577/ On Wed, 4 Feb 2026 at 16:48, Zhao, Guang via dev <[email protected]> wrote: > Hi Vaquar, > > Thank you for sharing your thoughts! > > I wonder about the relationship of KIP-1267 (Tiered Storage Cost > Attribution Metrics) to Diskless in general. > > Would Diskless be able to leverage the metrics proposed in KIP-1267, given > that they are mainly on the read path? I’d appreciate more details on this > in the KIP too. > > Thanks! > > > Cheers, > > Guang > > -- > > Guang Zhao, NetApp > > [email protected]<mailto:[email protected]> > > > > From: vaquar khan <[email protected]> > Date: Friday, 23 January 2026 at 3:13 am > To: [email protected] <[email protected]> > Subject: Strategic Alignment: Unifying KIP-1267 and the Cloud-Native > Roadmap > > EXTERNAL EMAIL - USE CAUTION when clicking links or attachments > > > > > Hi everyone, > > I’m opening this thread to help us get on the same page regarding the > recent "cloud-native" KIPs. > > In advancing *KIP-1267 (Tiered Storage Cost Attribution Metrics)*, I’ve > noticed we have several different initiatives—like diskless topics, remote > fetching, and better metrics—that are all moving in parallel. They are all > trying to solve the same problem: making Kafka cheaper and more elastic in > the cloud. However, they are currently disconnected. > > To ensure we build a cohesive platform rather than just a collection of > features, I propose we group these discussions into three main areas: > > *1. Cost Tracking (The Foundation)* We can't optimize what we can't > measure. *KIP-1267* ( > > https://urldefense.com/v3/__https://cwiki.apache.org/confluence/display/KAFKA/KIP-1267*3A*Tiered*Storage*Cost*Attribution*Metrics__;JSsrKysr!!Nhn8V6BzJA!RgzIg0Dsrd_zgbTJgD31AYGehPmAuhr6VTdY3HpKkNgkomvRSj32CRKpXLyPHNkxHIfZFkmTke85YrVcK2RwmQ$ > ) > gives us the granular metrics we need to actually bill users for storage > and API calls. This builds on the operational metrics from *KIP-963* ( > > https://urldefense.com/v3/__https://cwiki.apache.org/confluence/display/KAFKA/KIP-963*3A*Additional*metrics*in*Tiered*Storage__;JSsrKysr!!Nhn8V6BzJA!RgzIg0Dsrd_zgbTJgD31AYGehPmAuhr6VTdY3HpKkNgkomvRSj32CRKpXLyPHNkxHIfZFkmTke85YrUNcs31tA$ > ). > Without this layer, we cannot safely run the multi-tenant models we are > designing. > > *2. The Storage Decision* We need to decide which path to take for storage > disaggregation. Do we pursue the evolutionary path of *KIP-1176* ( > > https://urldefense.com/v3/__https://cwiki.apache.org/confluence/display/KAFKA/KIP-1176*3A*Tiered*Storage*for*Active*Log*Segment__;JSsrKysrKw!!Nhn8V6BzJA!RgzIg0Dsrd_zgbTJgD31AYGehPmAuhr6VTdY3HpKkNgkomvRSj32CRKpXLyPHNkxHIfZFkmTke85YrV9sD3Neg$ > ), > which keeps local disks for performance? Or do we go with the revolutionary > path of *KIP-1150* ( > > https://urldefense.com/v3/__https://cwiki.apache.org/confluence/display/KAFKA/KIP-1150*3A*Diskless*Topics__;JSsr!!Nhn8V6BzJA!RgzIg0Dsrd_zgbTJgD31AYGehPmAuhr6VTdY3HpKkNgkomvRSj32CRKpXLyPHNkxHIfZFkmTke85YrUiTZZyPg$ > ), > which removes disks entirely? This decision dictates our future > infrastructure and shouldn't be made in isolation. > > *3. Efficiency & Multi-Tenancy* We also have critical work happening on the > consumer side with *KIP-1248* ( > > https://urldefense.com/v3/__https://cwiki.apache.org/confluence/display/KAFKA/KIP-1248*3A*Broker*support*for*remote*tiered*storage*fetch*from*consumer__;JSsrKysrKysrKw!!Nhn8V6BzJA!RgzIg0Dsrd_zgbTJgD31AYGehPmAuhr6VTdY3HpKkNgkomvRSj32CRKpXLyPHNkxHIfZFkmTke85YrUzwFNn0Q$ > ) > and *KIP-1254* ( > > https://urldefense.com/v3/__https://cwiki.apache.org/confluence/display/KAFKA/KIP-1254*3A*Kafka*Consumer*Support*for*Remote*Tiered*Storage*Fetch__;JSsrKysrKysr!!Nhn8V6BzJA!RgzIg0Dsrd_zgbTJgD31AYGehPmAuhr6VTdY3HpKkNgkomvRSj32CRKpXLyPHNkxHIfZFkmTke85YrX0zViodA$ > < > https://urldefense.com/v3/__https://www.google.com/search?q=https:**Acwiki.apache.org*confluence*display*KAFKA*KIP-1254*253A*2BKafka*2BConsumer*2BSupport*2Bfor*2BRemote*2BTiered*2BStorage*2BFetch&authuser=1__;Ly8vLy8vJSUlJSUlJSUl!!Nhn8V6BzJA!RgzIg0Dsrd_zgbTJgD31AYGehPmAuhr6VTdY3HpKkNgkomvRSj32CRKpXLyPHNkxHIfZFkmTke85YrVX8F6l5g$ > >). > As we look toward *Virtual Clusters (KIP-1134)* ( > > https://urldefense.com/v3/__https://cwiki.apache.org/confluence/display/KAFKA/KIP-1134*3A*Multi-tenancy*in*Kafka*3A*Virtual*Clusters__;JSsrKyUrKw!!Nhn8V6BzJA!RgzIg0Dsrd_zgbTJgD31AYGehPmAuhr6VTdY3HpKkNgkomvRSj32CRKpXLyPHNkxHIfZFkmTke85YrWgeEZrJw$ > ) > and *Dynamic Controllers (KIP-853)* ( > > https://urldefense.com/v3/__https://cwiki.apache.org/confluence/display/KAFKA/KIP-853*3A*KRaft*Controller*Membership*Changes__;JSsrKys!!Nhn8V6BzJA!RgzIg0Dsrd_zgbTJgD31AYGehPmAuhr6VTdY3HpKkNgkomvRSj32CRKpXLyPHNkxHIfZFkmTke85YrVMFfTNxg$ > ), > the need for the rigorous cost tracking I’ve outlined in KIP-1267 becomes > even more urgent. > > I suggest we treat these KIPs as a single "Cloud-Native" capability set. > I’d like to discuss how *KIP-1267* can serve as the standard way to track > costs for these new architectures. > > Regards, > Viquar khan > > https://urldefense.com/v3/__https://www.linkedin.com/in/vaquar-khan-b695577/__;!!Nhn8V6BzJA!RgzIg0Dsrd_zgbTJgD31AYGehPmAuhr6VTdY3HpKkNgkomvRSj32CRKpXLyPHNkxHIfZFkmTke85YrU6ApbSDA$ > -- Regards, Vaquar Khan
