Thanks for the KIP Peter! Curious to see where this one goes, I think it's good to start a discussion around this though perhaps we'll need to split it up into more focused improvements as there's a lot bundled in this one idea!
A0. I'd like to see some folk that are more familiar with the broker implementation to chime in around the feasibility of implementing some of this. AFAIK, there's no capabilities that allow (for example) shifting resources between topics. Isolating that from a resource allocation perspective may be a huge lift, though certainly a valuable one. A1. With A0 in mind, I'm wondering what the benefit for making the QoS spec an open standard - it depends heavily both on the broker implementation and on how it's deployed (containerized? bare metal? k8s?). That makes what we can practically offer bundled with the default implementation limited. OTOH, I'm not sure whether users benefit from "open standards, free of vendor bias as much as possible" If the specification is customizable enough to allow for vendor specific extensions. A2. More a technical note, but the dynamic negotiation between producer and consumer seems to break a key abstraction of Kafka which is decoupling producers from consumers. That might work well if you have one consumer, but if you have multiple I imagine you wouldn't want one lagging to cause the producer to back up. I'll be following along, I'm sure there will be some good discussions around this! - Almog On Mon, May 12, 2025 at 4:47 PM Peter Corless <peter.corl...@startree.ai.invalid> wrote: > David Kjerrumgaard and I wrote up the following KIP for Kafka Quality of > Service (QoS). It would be a mechanism to describe desired behaviors and > actual capabilities of producers, clusters and consumers, and to allow them > to negotiate desired throughputs, latencies, data retention, and other > elements of data streaming. It would also provide instrumentality for > observability to measure actual performance to compare to desired > performance. > > Would love to hear frank and thoughtful feedback, as well as committers who > would be interested in working on implementation. > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1182%3A+Quality+of+Service+%28QoS%29+Framework > > -- > > [image: StarTree] <https://startree.ai> > Peter Corless > Director of Product Marketing > 650-906-3134 > Follow us: [image: LinkedIn] <https://www.linkedin.com/in/petercorless/ > >[image: > Twitter] <https://twitter.com/petercorless>[image: Slack] > <https://stree.ai/slack>[image: YouTube] > <https://youtube.com/StarTreeData>[image: > Calendly] <https://calendly.com/peter-corless/30min> > > [image: Save my spot for Real-Time Analytics Summit 2025] > < > https://rtasummit.startree.ai/?utm_source=referral&utm_medium=email&utm_campaign=signature > > >
