Hi, I was trying the single producer/consumer model via http://kafka.apache.org/documentation.html#quick-start. The code was from https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+Producer+Examplefor the producer. I'll try to repeat all the steps again in case I missed something but this was so frustrating and strange. I'll be so embarrassed if that's the case.
On Sun, Nov 10, 2013 at 7:51 PM, Priya Matpadi <priya.matp...@ecofactor.com>wrote: > If you have 5 partitions and 3 consumers in one consumer group, the > consumers will balance all 5 partitions such that 2 consumers will get data > from 2 partitions each and 1 consumer will get data from remaining 5th > partition. > If you have 3 partitions and 5 consumers in one consumer group, 3 consumers > will become leader for the 3 partitions(1 partition per consumer) and 2 > consumers will starve. Refer to > https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Group+Example. > > > On Sun, Nov 10, 2013 at 5:28 PM, Joe Stein <joe.st...@stealth.ly> wrote: > > > Do you have enough consumers representing the number of partitions you > > have? You should have one consumer within a consumer group for each > > partition... Think of it as threads, if you have 5 partitions you need 5 > > threads... if you only have 3 consumers(threads) then it is possible the > > data is on one of the other two partitions that has no one reading from > it. > > > > /******************************************* > > Joe Stein > > Founder, Principal Consultant > > Big Data Open Source Security LLC > > http://www.stealth.ly > > Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop> > > ********************************************/ > > > > > > On Sun, Nov 10, 2013 at 7:34 PM, S L <slouie.at.w...@gmail.com> wrote: > > > > > Thanks for everyone's reply and trying to help me. I think I got the > > test > > > producer program to work but I don't know why. > > > > > > Again, using the test code found at > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+Producer+Examplefor > > > the producer straight from the website didn't work. However, when I > > > changed the following line from > > > > > > KeyedMessage<String, String> data = new KeyedMessage<String, String>( > > > "page_visits", ip, msg); > > > to > > > KeyedMessage<String, String> data = new KeyedMessage<String, String>( > > > "page_visits", null, msg); > > > > > > all of a sudden, the consumer gets the messages from my test producer > > > program. Why does this work when the partition is equal to null and > not > > > work out of the box with the partition equal to ip. Is there a bug > with > > > the program that's posted in the link or is there some config that's > > > required that's not listed on the links? > > > > > > Thanks again for everyone's help. I really appreciate the community > > > replying and coming to help me. > > > > > > > > > On Fri, Nov 8, 2013 at 1:20 PM, Chris Bedford <ch...@buildlackey.com> > > > wrote: > > > > > > > I had some trouble with maven dependencies when i tried to get a > simple > > > > round trip test going. I worked past those and made my test > > available > > > > here: https://github.com/buildlackey/cep/tree/master/kafka > > > > > > > > it should run out of the box. > > > > > > > > -cb > > > > > > > > > > > > On Thu, Nov 7, 2013 at 5:35 PM, S L <slouie.at.w...@gmail.com> > wrote: > > > > > > > > > Hi, > > > > > > > > > > This might be a really, really simple question but how do I get my > > test > > > > > Kafka program working out of the box? I followed the directions > from > > > > > http://kafka.apache.org/documentation.html#quickstart. I started > > zk, > > > > the > > > > > server, the producer and consumer. I played with the producer, > > sending > > > > > msgs to the consumer, which I saw show up in its terminal. This > was > > to > > > > the > > > > > "page_visits" topic, as specified in the example program. > > > > > > > > > > I then went to > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+Producer+Exampleand > > > > > copied the files into netbeans and built it. I also added jar > files > > > > > kafka-assembly-0.8.0-deps.jar and kafka_2.8.0-0.8.0-beta1.jar to > make > > > the > > > > > build work if that makes a difference. > > > > > > > > > > Anyway, I then ran java -jar myjar.jar. I expected to see a > message > > > show > > > > > up in the consumer, but nothing shows up. I sent another msg from > > the > > > > > producer just to make sure the consumer was still connected, which > it > > > was > > > > > b/c the consumer received it. > > > > > > > > > > I'm at wits end. I know this is an extremely simple question and > > > > > everything looks identical to the links above and should be > working. > > > > > However, I'm not getting a message in the consumer. Much thanks > for > > > any > > > > > help you can give me. I'm about to ram my head into a wall. > > > > > > > > > > > > > > > > > > > > > -- > > > > Chris Bedford > > > > > > > > Founder & Lead Lackey > > > > Build Lackey Labs: http://buildlackey.com > > > > Go Grails!: http://blog.buildlackey.com > > > > > > > > > >
