Both use-case: Command queue for a streams job and work-queue in general seem reasonable. Thank you for explaining.
On Sun, Aug 12, 2018 at 6:34 AM, Matt Farmer <m...@frmr.me> wrote: > The work-queue use case is mostly how I see this being used, yes. > > In the most generic sense I can see its use in a situation where the > business dictates > that we have to guarantee quality of service for some set of low number of > messages while > there's some high number of messages being processed from a different > topic. > > For our pipelines at work, this would actually make it possible for us to > define a command > and control topic for some of our streams applications. We occasionally > have to change the > behavior of our streams in reaction to system issues or, occasionally, > user's abusing the > system and creating a bunch of garbage data that we'd like to skip over. > Today, we have > to either define that behavior in a config setting and restart the > application (which is what > we currently do) or implement some sort of API external to streams that the > stream pulls > state from. > > With a CnC topic, we could interleave these into the normal stream > processing flow and > instruct it to alter a state store, for example, with the criteria for > records to be dropped > without introducing other libraries or having to manually synchronize with > external state. > > On Wed, Aug 8, 2018 at 10:11 PM Gwen Shapira <g...@confluent.io> wrote: > > > Can you guys spell it out for me? I just don't really see when I want to > > subscribe to two topics but not get events from both at the same time. > > Is this a work-queue type pattern? > > > > On Wed, Aug 8, 2018 at 6:10 PM, Matt Farmer <m...@frmr.me> wrote: > > > > > Oh, almost forgot, thanks for the KIP - I can see this being a very > > useful > > > addition. :) > > > > > > On Wed, Aug 8, 2018 at 9:09 PM Matt Farmer <m...@frmr.me> wrote: > > > > > > > Is it worth spelling out explicitly what the behavior is when two > > topics > > > > have the same priority? I'm a bit fuzzy on how we choose what topics > to > > > > consume from right now, if I'm being honest, so it could be useful to > > > > outline the current behavior in the background and to spell out how > > that > > > > would change (or if it would change) when two topics are given the > same > > > > priority. > > > > > > > > Also, how does this play with max.poll.records? Does the consumer > read > > > > from all the topics in priority order until we've hit the number of > > > records > > > > or the poll timeout? Or does it immediately return the high priority > > > > records without pulling low priority records? > > > > > > > > On Wed, Aug 8, 2018 at 8:39 PM <n...@afshartous.com> wrote: > > > > > > > >> > > > >> Hi All, > > > >> > > > >> Calling for a vote on KIP-349 > > > >> > > > >> > > > >> https://cwiki.apache.org/confluence/display/KAFKA/KIP- > > > 349%3A+Priorities+for+Source+Topics > > > >> < > > > >> https://cwiki.apache.org/confluence/display/KAFKA/KIP- > > > 349:+Priorities+for+Source+Topics > > > >> > > > > >> > > > >> Cheers, > > > >> -- > > > >> Nick > > > >> > > > >> > > > >> > > > >> > > > > > > > > > > > -- > > *Gwen Shapira* > > Product Manager | Confluent > > 650.450.2760 | @gwenshap > > Follow us: Twitter <https://twitter.com/ConfluentInc> | blog > > <http://www.confluent.io/blog> > > > -- *Gwen Shapira* Product Manager | Confluent 650.450.2760 | @gwenshap Follow us: Twitter <https://twitter.com/ConfluentInc> | blog <http://www.confluent.io/blog>
