Do you have enough consumers representing the number of partitions you have? You should have one consumer within a consumer group for each partition... Think of it as threads, if you have 5 partitions you need 5 threads... if you only have 3 consumers(threads) then it is possible the data is on one of the other two partitions that has no one reading from it.
/******************************************* Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop> ********************************************/ On Sun, Nov 10, 2013 at 7:34 PM, S L <slouie.at.w...@gmail.com> wrote: > Thanks for everyone's reply and trying to help me. I think I got the test > producer program to work but I don't know why. > > Again, using the test code found at > https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+Producer+Examplefor > the producer straight from the website didn't work. However, when I > changed the following line from > > KeyedMessage<String, String> data = new KeyedMessage<String, String>( > "page_visits", ip, msg); > to > KeyedMessage<String, String> data = new KeyedMessage<String, String>( > "page_visits", null, msg); > > all of a sudden, the consumer gets the messages from my test producer > program. Why does this work when the partition is equal to null and not > work out of the box with the partition equal to ip. Is there a bug with > the program that's posted in the link or is there some config that's > required that's not listed on the links? > > Thanks again for everyone's help. I really appreciate the community > replying and coming to help me. > > > On Fri, Nov 8, 2013 at 1:20 PM, Chris Bedford <ch...@buildlackey.com> > wrote: > > > I had some trouble with maven dependencies when i tried to get a simple > > round trip test going. I worked past those and made my test available > > here: https://github.com/buildlackey/cep/tree/master/kafka > > > > it should run out of the box. > > > > -cb > > > > > > On Thu, Nov 7, 2013 at 5:35 PM, S L <slouie.at.w...@gmail.com> wrote: > > > > > Hi, > > > > > > This might be a really, really simple question but how do I get my test > > > Kafka program working out of the box? I followed the directions from > > > http://kafka.apache.org/documentation.html#quickstart. I started zk, > > the > > > server, the producer and consumer. I played with the producer, sending > > > msgs to the consumer, which I saw show up in its terminal. This was to > > the > > > "page_visits" topic, as specified in the example program. > > > > > > I then went to > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+Producer+Exampleand > > > copied the files into netbeans and built it. I also added jar files > > > kafka-assembly-0.8.0-deps.jar and kafka_2.8.0-0.8.0-beta1.jar to make > the > > > build work if that makes a difference. > > > > > > Anyway, I then ran java -jar myjar.jar. I expected to see a message > show > > > up in the consumer, but nothing shows up. I sent another msg from the > > > producer just to make sure the consumer was still connected, which it > was > > > b/c the consumer received it. > > > > > > I'm at wits end. I know this is an extremely simple question and > > > everything looks identical to the links above and should be working. > > > However, I'm not getting a message in the consumer. Much thanks for > any > > > help you can give me. I'm about to ram my head into a wall. > > > > > > > > > > > -- > > Chris Bedford > > > > Founder & Lead Lackey > > Build Lackey Labs: http://buildlackey.com > > Go Grails!: http://blog.buildlackey.com > > >
