If you are using new Java producer, reorder could happen if max.inflight.requests.per.connection is set to > 1 and retries are enabled - which are both default settings.
Can you set max.in.flight.requests.per.connection to 1 and see if this solve the issue? Jiangjie (Becket) Qin On 5/12/15, 12:57 PM, "Scott Chapman" <sc...@woofplanet.com> wrote: >We are using the Java producer API (0.8.2.1 if I am not mistaken). We are >using producer type of sync though. > >On Tue, May 12, 2015 at 3:50 PM Magnus Edenhill <mag...@edenhill.se> >wrote: > >> Hi Scott, >> >> what producer client are you using? >> >> Reordering is possible in async producers in the case of temporary >>broker >> failures >> and the combination of request.required.acks != 0 and retries > 0. >> >> Consider the case where a producer has 20 messages in-flight to the >>broker, >> out of those >> messages # 1-10 fails due to some temporary failure (?) on the broker >>side, >> but message # 11-20 are accepted. >> When the producer receives error results from the broker for message # >>1-10 >> it will try to resend >> these 10 failed messages, that are now accepted, causing them to end up >> after message #20 in the log - thus reordered. >> >> This failure scenario should be rather rare though. >> >> >> Regards, >> Magnus >> >> 2015-05-12 20:18 GMT+02:00 Scott Chapman <sc...@woofplanet.com>: >> >> > We are basically using kafka as a transport mechanism for multi-line >>log >> > files. >> > >> > So, for this we are using single partition topics (with a replica for >> good >> > measure) writing to a multi-broker cluster. >> > >> > Our producer basically reads a file line-by-line (as it is being >>written >> > to) and publishes each line as a message to the topic. We are also >> writing >> > as quickly as we can (not waiting for ACK). >> > >> > What I am seeing is occasionally the messages in the topic appear to >>be >> > slightly out of order when compared to the source file they were based >> on. >> > >> > I am wonder if this might happen when the producer switches brokers >> because >> > we are not waiting for the ACK before continuing to write. >> > >> > Does this make any sense?? >> > >> > Thanks in advance! >> > >> > -Scott >> > >>
