In thinking on it, another solution for this is another consumer external to the stream - but then we run into timing issues and complexity with using a state store as the storage of record. :/ On Sun, Aug 12, 2018 at 9:34 AM Matt Farmer <m...@frmr.me> wrote:
> The work-queue use case is mostly how I see this being used, yes. > > In the most generic sense I can see its use in a situation where the > business dictates > that we have to guarantee quality of service for some set of low number of > messages while > there's some high number of messages being processed from a different > topic. > > For our pipelines at work, this would actually make it possible for us to > define a command > and control topic for some of our streams applications. We occasionally > have to change the > behavior of our streams in reaction to system issues or, occasionally, > user's abusing the > system and creating a bunch of garbage data that we'd like to skip over. > Today, we have > to either define that behavior in a config setting and restart the > application (which is what > we currently do) or implement some sort of API external to streams that > the stream pulls > state from. > > With a CnC topic, we could interleave these into the normal stream > processing flow and > instruct it to alter a state store, for example, with the criteria for > records to be dropped > without introducing other libraries or having to manually synchronize with > external state. > > On Wed, Aug 8, 2018 at 10:11 PM Gwen Shapira <g...@confluent.io> wrote: > >> Can you guys spell it out for me? I just don't really see when I want to >> subscribe to two topics but not get events from both at the same time. >> Is this a work-queue type pattern? >> >> On Wed, Aug 8, 2018 at 6:10 PM, Matt Farmer <m...@frmr.me> wrote: >> >> > Oh, almost forgot, thanks for the KIP - I can see this being a very >> useful >> > addition. :) >> > >> > On Wed, Aug 8, 2018 at 9:09 PM Matt Farmer <m...@frmr.me> wrote: >> > >> > > Is it worth spelling out explicitly what the behavior is when two >> topics >> > > have the same priority? I'm a bit fuzzy on how we choose what topics >> to >> > > consume from right now, if I'm being honest, so it could be useful to >> > > outline the current behavior in the background and to spell out how >> that >> > > would change (or if it would change) when two topics are given the >> same >> > > priority. >> > > >> > > Also, how does this play with max.poll.records? Does the consumer read >> > > from all the topics in priority order until we've hit the number of >> > records >> > > or the poll timeout? Or does it immediately return the high priority >> > > records without pulling low priority records? >> > > >> > > On Wed, Aug 8, 2018 at 8:39 PM <n...@afshartous.com> wrote: >> > > >> > >> >> > >> Hi All, >> > >> >> > >> Calling for a vote on KIP-349 >> > >> >> > >> >> > >> https://cwiki.apache.org/confluence/display/KAFKA/KIP- >> > 349%3A+Priorities+for+Source+Topics >> > >> < >> > >> https://cwiki.apache.org/confluence/display/KAFKA/KIP- >> > 349:+Priorities+for+Source+Topics >> > >> > >> > >> >> > >> Cheers, >> > >> -- >> > >> Nick >> > >> >> > >> >> > >> >> > >> >> > >> >> >> >> -- >> *Gwen Shapira* >> Product Manager | Confluent >> 650.450.2760 | @gwenshap >> Follow us: Twitter <https://twitter.com/ConfluentInc> | blog >> <http://www.confluent.io/blog> >> >
