An auto-increment index can be assigned to a message as a key when it is being published. The consumer can monitor this index when receiving. If the expected message does not show up, buffer all received messages in a hashtable (use index as hash key) until it is received. Then handle all messages in the hashtable.
Regards, Libo -----Original Message----- From: Jun Rao [mailto:jun...@gmail.com] Sent: Friday, August 23, 2013 11:20 AM To: users@kafka.apache.org Subject: Re: Producer message ordering problem Ross, This is a general issue with resending. Since resending is typically done on a new socket, essentially new messages are sent from a new instance of producer. So, there is no easy way to ensure that the new messages are ordered behind the ones sent by the old instance of the producer. So 0.8 will have similar issues. It may be possible to add some sort of per client sequence id and track that in the broker. But this may not be trivial and will need more thoughts. Thanks, Jun On Thu, Aug 22, 2013 at 9:32 PM, Ross Black <ross.w.bl...@gmail.com> wrote: > Hi, > > I am using Kafka 0.7.1, and using the low-level SyncProducer to send > messages to a *single* partition from a *single* thread. > The client sends messages that contain sequential numbers so it is > obvious at the consumer when message order is shuffled. > I have noticed that messages can be saved out-or-order by Kafka when > there are connection problems, and am looking for possible solutions > (I think I already know the cause). > > The client sends messages in a retry loop so that it will wait for a > short period and then retry to send on any IO errors. In > SyncProducer, any IOException triggers a disconnect. Next time send > is called a new connection is established. I believe that it is this > disconnect/reconnect cycle that can cause messages to be saved to the > kafka log in a different order to that of the client. > > I had previously had the same sort of issue with > reconnect.interval/time, which was fixed by disabling those reconnect > settings. > > http://mail-archives.apache.org/mod_mbox/kafka-users/201305.mbox/%3CCA > M%2BbZhjssxmUhn_L%3Do0bGsD7PAXFGQHRpOKABcLz29vF3cNOzA%40mail.gmail.com > %3E > > Is there anything in 0.7 that would allow me to solve this problem? > The only option I can see at the moment is to not perform retries. > > Does 0.8 handle this issue any differently? > > Thanks, > Ross >
