Thank you Chia-chun,Joe and Jagat. I am not using any custom partitioner logic. Here is what I observed when I ran kafka on 4 nodes with the following structure:
1. Each node has a producer, consumer and a broker (that contains one partition of my topic) and one of the machine has the Zookeeper too. 2. Producer in each node sends 10000 messages to my topic. 3. I observed that consumer in all 4 nodes gets some messages and some times only 2 nodes receive messages and one doesn't and sometimes only one node receives messages and 3 doesnt receive any messages. So according to my observation, producer is sending messages to random partition. Am I correct? Thank you Sandeep On Thu, Nov 13, 2014 at 9:34 PM, Joe Stein <joe.st...@stealth.ly> wrote: > Yup, sounds like > > https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Whyisdatanotevenlydistributedamongpartitionswhenapartitioningkeyisnotspecified > ? > > This should go away with 0.8.2 with the default partions now being 1 =8^) > with auto create topics. > > /******************************************* > Joe Stein > Founder, Principal Consultant > Big Data Open Source Security LLC > http://www.stealth.ly > Twitter: @allthingshadoop > ********************************************/ > On Nov 13, 2014 8:34 PM, "Chia-Chun Shih" <chiachun.s...@gmail.com> wrote: > > > Hi Palur, > > > > When producing messages, did you specify a key in your KeyedMessage? If > > not, producer will send all messages to ONE randomly selected partition > and > > stick to this partition for 10 minutes by default. > > > > regards, > > Chia-Chun > > > > 2014-11-14 7:19 GMT+08:00 Jagat Singh <jagatsi...@gmail.com>: > > > > > It would be worth reading once the consumer section from the > > documentation. > > > > > > https://kafka.apache.org/documentation.html > > > > > > > > > > > > On Fri, Nov 14, 2014 at 10:09 AM, Palur Sandeep <psand...@hawk.iit.edu > > > > > wrote: > > > > > > > Yes, they are on the same consumer group, but I have two partitions. > > > > > > > > On Thu, Nov 13, 2014 at 5:04 PM, Jagat Singh <jagatsi...@gmail.com> > > > wrote: > > > > > > > > > Are both of them in same Consumer Group? > > > > > > > > > > On Fri, Nov 14, 2014 at 9:12 AM, Palur Sandeep < > > psand...@hawk.iit.edu> > > > > > wrote: > > > > > > > > > > > Dear Developers, > > > > > > > > > > > > I am 2nd year masters student at IIT. I am using Kafka for one of > > my > > > > > > research projects.My question is the following: > > > > > > > > > > > > 1. I have a producer, consumer and a broker(that contains 1st > > > partition > > > > > of > > > > > > my topic) on node1 > > > > > > 2. I have a producer, consumer, zookeeper and a broker(that > > contains > > > > 2nd > > > > > > partition of my topic) on node2 > > > > > > 3. Here comes my problem: though I have two partitions only one > > > > consumer > > > > > > pulls messages and the other one is always idle. > > > > > > > > > > > > What is that I can do to keep both of my consumer busy? > > > > > > > > > > > > Thank you > > > > > > > > > > > > > > > > > > -- > > > > > > Regards, > > > > > > Sandeep Palur > > > > > > Data-Intensive Distributed Systems Laboratory, CS/IIT > > > > > > Department of Computer Science, Illinois Institute of Technology > > > (IIT) > > > > > > Phone : 312-647-9833 > > > > > > Email : psand...@hawk.iit.edu <sraja...@hawk.iit.edu> > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > Regards, > > > > Sandeep Palur > > > > Data-Intensive Distributed Systems Laboratory, CS/IIT > > > > Department of Computer Science, Illinois Institute of Technology > (IIT) > > > > Phone : 312-647-9833 > > > > Email : psand...@hawk.iit.edu <sraja...@hawk.iit.edu> > > > > > > > > > > -- Regards, Sandeep Palur Data-Intensive Distributed Systems Laboratory, CS/IIT Department of Computer Science, Illinois Institute of Technology (IIT) Phone : 312-647-9833 Email : psand...@hawk.iit.edu <sraja...@hawk.iit.edu>
