Hi All, New to kafka and would love some feedback on how to think about a solution for this kind of flow:
So, sequencing must be maintained for events of type X:Y:PAYLOAD Where X is a grouping mechanism and all X must be processed sequentially e.g. account number, (ok i use key here to keep these in sequence) But Y is a secondary grouping and distribution variable, that will fan out to competing consumers for each type of Y e.g. where y=A it goes to a topic of competing consumers and where y=B it goes out to a different topic of competing consumers, etc..Each competing consumer group is configured to handle only Ys of its type. All Xs must consumed sequentially, across distributed consumer groups even those fanned out with a subkey of Y How do I keep it sequential processing FIFO across? I know I can use a key to sequence by X in a topic partition for sequential processing, but it breaks down from there for me as I also need to sequence by Y. Is there a good built in way to handle this type of problem? Am I thinking about it wrong? I hope this is somewhat clear? thanks for any input! looks something like this maybe: [image: scheme1.png] -- Thanks, -Andre
