If you're not using your own partitioning logic, messages are partitioned randomly. This is the current default behavior I believe.
On Fri, Nov 14, 2014 at 12:01 PM, Palur Sandeep <psand...@hawk.iit.edu> wrote: > Thank you Chia-chun,Joe and Jagat. > > I am not using any custom partitioner logic. Here is what I observed when I > ran kafka on 4 nodes with the following structure: > > 1. Each node has a producer, consumer and a broker (that contains one > partition of my topic) and one of the machine has the Zookeeper too. > 2. Producer in each node sends 10000 messages to my topic. > 3. I observed that consumer in all 4 nodes gets some messages and some > times only 2 nodes receive messages and one doesn't and sometimes only one > node receives messages and 3 doesnt receive any messages. > > So according to my observation, producer is sending messages to random > partition. > > Am I correct? > > Thank you > Sandeep > > > > > > > On Thu, Nov 13, 2014 at 9:34 PM, Joe Stein <joe.st...@stealth.ly> wrote: > > > Yup, sounds like > > > > > https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Whyisdatanotevenlydistributedamongpartitionswhenapartitioningkeyisnotspecified > > ? > > > > This should go away with 0.8.2 with the default partions now being 1 =8^) > > with auto create topics. > > > > /******************************************* > > Joe Stein > > Founder, Principal Consultant > > Big Data Open Source Security LLC > > http://www.stealth.ly > > Twitter: @allthingshadoop > > ********************************************/ > > On Nov 13, 2014 8:34 PM, "Chia-Chun Shih" <chiachun.s...@gmail.com> > wrote: > > > > > Hi Palur, > > > > > > When producing messages, did you specify a key in your KeyedMessage? If > > > not, producer will send all messages to ONE randomly selected partition > > and > > > stick to this partition for 10 minutes by default. > > > > > > regards, > > > Chia-Chun > > > > > > 2014-11-14 7:19 GMT+08:00 Jagat Singh <jagatsi...@gmail.com>: > > > > > > > It would be worth reading once the consumer section from the > > > documentation. > > > > > > > > https://kafka.apache.org/documentation.html > > > > > > > > > > > > > > > > On Fri, Nov 14, 2014 at 10:09 AM, Palur Sandeep < > psand...@hawk.iit.edu > > > > > > > wrote: > > > > > > > > > Yes, they are on the same consumer group, but I have two > partitions. > > > > > > > > > > On Thu, Nov 13, 2014 at 5:04 PM, Jagat Singh <jagatsi...@gmail.com > > > > > > wrote: > > > > > > > > > > > Are both of them in same Consumer Group? > > > > > > > > > > > > On Fri, Nov 14, 2014 at 9:12 AM, Palur Sandeep < > > > psand...@hawk.iit.edu> > > > > > > wrote: > > > > > > > > > > > > > Dear Developers, > > > > > > > > > > > > > > I am 2nd year masters student at IIT. I am using Kafka for one > of > > > my > > > > > > > research projects.My question is the following: > > > > > > > > > > > > > > 1. I have a producer, consumer and a broker(that contains 1st > > > > partition > > > > > > of > > > > > > > my topic) on node1 > > > > > > > 2. I have a producer, consumer, zookeeper and a broker(that > > > contains > > > > > 2nd > > > > > > > partition of my topic) on node2 > > > > > > > 3. Here comes my problem: though I have two partitions only one > > > > > consumer > > > > > > > pulls messages and the other one is always idle. > > > > > > > > > > > > > > What is that I can do to keep both of my consumer busy? > > > > > > > > > > > > > > Thank you > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > Regards, > > > > > > > Sandeep Palur > > > > > > > Data-Intensive Distributed Systems Laboratory, CS/IIT > > > > > > > Department of Computer Science, Illinois Institute of > Technology > > > > (IIT) > > > > > > > Phone : 312-647-9833 > > > > > > > Email : psand...@hawk.iit.edu <sraja...@hawk.iit.edu> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > Regards, > > > > > Sandeep Palur > > > > > Data-Intensive Distributed Systems Laboratory, CS/IIT > > > > > Department of Computer Science, Illinois Institute of Technology > > (IIT) > > > > > Phone : 312-647-9833 > > > > > Email : psand...@hawk.iit.edu <sraja...@hawk.iit.edu> > > > > > > > > > > > > > > > > > > -- > Regards, > Sandeep Palur > Data-Intensive Distributed Systems Laboratory, CS/IIT > Department of Computer Science, Illinois Institute of Technology (IIT) > Phone : 312-647-9833 > Email : psand...@hawk.iit.edu <sraja...@hawk.iit.edu> > -- -- Sharninder
