Also check out writing a custom partitioner. You could figure out the number of partitions and then use a custom partitioner to write a stop message to each.
I would consider looking into not needing this functionality at all if possible. Dave > On May 29, 2016, at 15:18, Tauzell, Dave <dave.tauz...@surescripts.com> wrote: > > If you use keys it might be possible to write a "no more messages" message to > each partition. > > Dave. > >> On May 29, 2016, at 01:51, David Olsen <davidolsen4...@gmail.com> wrote: >> >> I understand that the reception of Kafka messages may be out of >> order[1][2]. But I have a multiple producers/ consumers scenario where I >> would need to know when the last record is processed at the producer side, >> so once consumers process e.g. NoMoreMessage, the entire system can react >> such as stopping the consumers, proceeding to the next step, and so on. But >> due to the nature of distributed system, there is no guaranteed that >> NoMoreMessage issued at the producer side would be the last message >> processed by consumers. So I need pieces of advice if any strategies I can >> exploit in this scenario with Kafka? >> >> I appreciate any suggestions. >> >> [1]. >> https://www.quora.com/Is-it-possible-to-consume-kafka-message-offset-based-on-timestamp >> [2]. >> http://stackoverflow.com/questions/35525356/kafka-multiple-producers-writing-to-same-topic-and-order-of-message-is-importa?rq=1 > This e-mail and any files transmitted with it are confidential, may contain > sensitive information, and are intended solely for the use of the individual > or entity to whom they are addressed. If you have received this e-mail in > error, please notify the sender by reply e-mail immediately and destroy all > copies of the e-mail and any attachments.
