Hi Jun, This is my original concern: Assume the number of consumer threads is the same as the number of partitions of a topic. And we want to leverage the fact that messages in a partition are ordered. But when starting multiple consumer threads or rebalancing, it is possible that one consumer thread will temporarily consume two or more partitions.
After thinking it over, I realize this can be solved in the consumer thread as long as we never assume one consumer thread will only consume one partition. Regards, Libo -----Original Message----- From: Jun Rao [mailto:jun...@gmail.com] Sent: Thursday, August 22, 2013 12:01 AM To: users@kafka.apache.org Subject: Re: ordering Actually, I am not sure if I understand the trouble that you mentioned. Could you elaborate that a bit more? Thanks, Jun On Wed, Aug 21, 2013 at 12:30 PM, Yu, Libo <libo...@citi.com> wrote: > Hi, > > This is from kafka faq: > > > * Each partition is not consumed by more than one consumer > thread/process in each consumer group. This allows to have each > process consume in a single threaded fashion to guarantee ordering to > the consumer within the partition (if we split up a partition of > ordered messages and handed them out to multiple consumers even though > the messages were stored in order they would be processed out of order at > times). > > Is this doable? > > Say a topic has 3 partitions. And there are 3 consumer processes in a > consumer group each of which has a single thread. > When we start the 3 processes, when the first process is up, it will > consume all 3 partitions. When the second process is up, > 1 process consumes 2 partitions and 1 consumes 1 partition. Only when > the third process is up, will each process consume One partition. This > will cause trouble. There seems to be no way to bound a stream to one > partition. > > > Regards, > > Libo > >
