kinda: A:X:FOO B:X:FOO B:Y:BAR B:Z:BAZ A:X:FOO and B:X:FOO can be processed in parallel, but B:Z:BAZ must be processed after B:Y:BAR and after B:X:FOO where A and B go to some partition for sequencing, but then "X", "Y" and "Z" go to a different sub topic and partition for processing.
sequencing must maintain across all Seq->A:X:FOO Seq->B:X:FOO, B:Y:BAR, B:Z:BAZ On Wed, Sep 2, 2020 at 5:31 PM Tauzell, Dave <dave.tauz...@surescripts.com> wrote: > So if the stream is: > > A:1:FOO > A:3:BAR > A:3:BAZ > > Then A:3* must be processed after A:1 but A:3:BAR and A:3:BAZ can be > processed in any order? > > I don’t think there is a way to do that with topics. > -Dave > > > From: Andre Mermegas <and...@gmail.com> > Reply-To: "users@kafka.apache.org" <users@kafka.apache.org> > Date: Wednesday, September 2, 2020 at 4:06 PM > To: "users@kafka.apache.org" <users@kafka.apache.org> > Subject: [External] key and subkey sequential processing across competing > consumer groups? > > Hi All, > > New to kafka and would love some feedback on how to think about a solution > for this kind of flow: > > So, sequencing must be maintained for events of type X:Y:PAYLOAD > > Where X is a grouping mechanism and all X must be processed sequentially > e.g. account number, (ok i use key here to keep these in sequence) > > But Y is a secondary grouping and distribution variable, that will fan out > to competing consumers for each type of Y > > e.g. where y=A it goes to a topic of competing consumers and where y=B it > goes out to a different topic of competing consumers, etc..Each competing > consumer group is configured to handle only Ys of its type. All Xs must > consumed sequentially, across distributed consumer groups even those fanned > out with a subkey of Y > > How do I keep it sequential processing FIFO across? I know I can use a key > to sequence by X in a topic partition for sequential processing, but it > breaks down from there for me as I also need to sequence by Y. Is there a > good built in way to handle this type of problem? Am I thinking about it > wrong? > > I hope this is somewhat clear? thanks for any input! > > looks something like this maybe: > > [cid:ii_kelvcjxy1] > > -- > Thanks, > -Andre > This e-mail and any files transmitted with it are confidential, may > contain sensitive information, and are intended solely for the use of the > individual or entity to whom they are addressed. If you have received this > e-mail in error, please notify the sender by reply e-mail immediately and > destroy all copies of the e-mail and any attachments. > -- Thanks, -Andre
