Hi, We have a situation where we are streaming data in a Kafka topic to be processed. The topic has multiple partitions. From time to time we need to add control messages to the stream and this needs to be coherent with the data steam (e.g. "replay from point X to this new location"). Because of the way Kafka partitions it seems hard to get these control messages to every consumer, no matter how they have been assigned, in the right place in the queues. The way I see it there are two options:
1) Have the producer put these control messages on every partition. We could get the number of partitions and manually post the same message on every partition, which will guarantee it gets read in the right place. 2) Reserve a special partition (e.g. 0) to be a control one, and have all consumers subscribe to that one. I don't think this gives the guarantees we want though. The other option is to use a separate queue for these, but then we'd need some special logic to ensure they have their effect at the right point in the queue. Has anyone else implemented anything like this? Am I barking up the wrong tree? Thanks, Ben Ben Young Principal Software Engineer - Risk Risk & Compliance T: +44 (0) 1865 578140 E: ben.yo...@fisglobal.com FIS | Empowering the Financial World FIS Systems Limited * Registered in England and Wales No. 1937159 * Registered Office: Level 39, 25 Canada Square, London E14 5LQ The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you.
