Hello Mathieu, Is the web service module and the background component sharing the same producer client to send messages, and are they sending messages to the same topic?
If answer is yet to both, then ordering should be preserved, since the producer.send() call will append the messages to the partitions in order, and with default partitioner messages with the same key will go to the same partition, so ordering is preserved. If you are using separate producers, however, then due to batching the ordering may be modified. Guozhang On Tue, Sep 27, 2016 at 2:50 AM, Eno Thereska <eno.there...@gmail.com> wrote: > Hi Mathieu, > > If the messages are sent asynchronously, then what you're observing is > indeed right. There is no guarantee that the first will arrive at the > destination first. > Perhaps you can try sending them synchronously (i.e., wait until the first > one is received, before sending the second). That might slow down your > pipeline, but perhaps that's acceptable? > > Eno > > > > On 27 Sep 2016, at 04:26, Mathieu Fenniak <mathieu.fenn...@replicon.com> > wrote: > > > > Hey Apache Users, > > > > I'm working on a web application that has a web service component, and a > > background processor component. Both applications will send messages to > > the same Kafka topic as an object is manipulated. > > > > In some cases, a web service call in the service component will send a > > message to Kafka saying key K has state S1, then trigger a background > > operation, and then the background component will send a message to Kafka > > saying key K has state S2. However, I'm finding that the topic ends up > > occasionally having a message K/S2 followed by K/S1, rather than the > other > > way around. As both producers in the web service call and the background > > processor send messages asynchronously with librdkafka, I believe this > is a > > relatively simple race condition where messages just aren't coming in > like > > I'd like them to. > > > > In a consuming Kafka Streams application, I'd be creating a KTable of > this > > topic. What approaches can I take to ensure the the KTable will end up > > with K/S2 as the state for K, rather than the stale-er K/S1? > > > > Would KS reorder messages if they had ordered & coordinated timestamps? > If > > so, how much leeway would it have for S2 being delivered before S1? (I > > believe librdkafka 0.9.1 doesn't support sending create-time in messages, > > which makes this is a bit more painful.) > > > > Any other approaches that are worth exploring? > > > > Thanks for any thoughts, > > > > Mathieu > > -- -- Guozhang
