The work-queue use case is mostly how I see this being used, yes. In the most generic sense I can see its use in a situation where the business dictates that we have to guarantee quality of service for some set of low number of messages while there's some high number of messages being processed from a different topic.
For our pipelines at work, this would actually make it possible for us to define a command and control topic for some of our streams applications. We occasionally have to change the behavior of our streams in reaction to system issues or, occasionally, user's abusing the system and creating a bunch of garbage data that we'd like to skip over. Today, we have to either define that behavior in a config setting and restart the application (which is what we currently do) or implement some sort of API external to streams that the stream pulls state from. With a CnC topic, we could interleave these into the normal stream processing flow and instruct it to alter a state store, for example, with the criteria for records to be dropped without introducing other libraries or having to manually synchronize with external state. On Wed, Aug 8, 2018 at 10:11 PM Gwen Shapira <g...@confluent.io> wrote: > Can you guys spell it out for me? I just don't really see when I want to > subscribe to two topics but not get events from both at the same time. > Is this a work-queue type pattern? > > On Wed, Aug 8, 2018 at 6:10 PM, Matt Farmer <m...@frmr.me> wrote: > > > Oh, almost forgot, thanks for the KIP - I can see this being a very > useful > > addition. :) > > > > On Wed, Aug 8, 2018 at 9:09 PM Matt Farmer <m...@frmr.me> wrote: > > > > > Is it worth spelling out explicitly what the behavior is when two > topics > > > have the same priority? I'm a bit fuzzy on how we choose what topics to > > > consume from right now, if I'm being honest, so it could be useful to > > > outline the current behavior in the background and to spell out how > that > > > would change (or if it would change) when two topics are given the same > > > priority. > > > > > > Also, how does this play with max.poll.records? Does the consumer read > > > from all the topics in priority order until we've hit the number of > > records > > > or the poll timeout? Or does it immediately return the high priority > > > records without pulling low priority records? > > > > > > On Wed, Aug 8, 2018 at 8:39 PM <n...@afshartous.com> wrote: > > > > > >> > > >> Hi All, > > >> > > >> Calling for a vote on KIP-349 > > >> > > >> > > >> https://cwiki.apache.org/confluence/display/KAFKA/KIP- > > 349%3A+Priorities+for+Source+Topics > > >> < > > >> https://cwiki.apache.org/confluence/display/KAFKA/KIP- > > 349:+Priorities+for+Source+Topics > > >> > > > >> > > >> Cheers, > > >> -- > > >> Nick > > >> > > >> > > >> > > >> > > > > > > -- > *Gwen Shapira* > Product Manager | Confluent > 650.450.2760 | @gwenshap > Follow us: Twitter <https://twitter.com/ConfluentInc> | blog > <http://www.confluent.io/blog> >
