It may be hard to reason about ordering across 1400 partitions. Could you use the SimpleConsumerShell to consume messages from 1 partition and see if messages are ordered?
Thanks, Jun On Fri, Mar 21, 2014 at 1:04 AM, Tom Amon <ta46...@gmail.com> wrote: > Hi All, > > I have a question regarding ordering of consumed messages. We timestamp our > messages and send them into Kafka in order. I wrote a simple consumer that > simply consumes the messages and prints out the timestamp. I see messages > for all seven days worth of date being consumed at once. > > Our setup: > Kafka 0.8 > 5 Kafka brokers > 1400 partitions > > The consumer has 10 threads, simply connects, consumes and prints > timestamps. It is set to the "smallest" offset so that it reads from the > beginning. There are many millions of messages so I think I can rule out > some partitions not having messages for certain days as the cause. I know > that Kafka doesn't guarantee ordering across partitions but I would assume > that with this volume of messages I would see the timestamps for the first > day, followed by the second day, etc. Instead I see them all print at once. > > Any ideas what I might be doing wrong? >
