If a consumer comes from a new consumer group, it will by default consume from the log end. In you first scenario, because you started consumer after producing data, when the consumer starts it will consume from the latest. So you will not be able to see the messages even though they are in the broker. In your second scenario, because consumer will commit the offsets periodically and during rebalance. The first run of consumer will commit offset to be 0. Then you produced some data and run the second consumer. Because the second run of consumer uses the same consumer group, it will pick up the committed offset and consume from 0. In this case you will see the produced messages. If you use a different consumer group in your second run of consumer, you will not be able to see the messages because the offset will be reset to latest like in the first scenario.
-Jiangjie Qin On 2/24/15, 1:56 PM, "Yang" <teddyyyy...@gmail.com> wrote: >somehow I have to let the consumer run once BEFORE the messages are >produced. then run consumer after messages are produced, then it works: > > >export topic=ttnew21 > sudo -u kafka bin/kafka-topics.sh --zookeeper localhost:7181 --create >--topic $topic --partitions 1 --replication-factor 1 >echo "first try" > sudo -u kafka bin/kafka-console-consumer.sh --zookeeper localhost:7181 >--consumer-timeout-ms 1000 --topic $topic -group fffffxx > echo 'asdfasdfsadffadsfasf' | sudo -u kafka >bin/kafka-console-producer.sh >--broker-list localhost:9092 --topic $topic >echo "second try" > sudo -u kafka bin/kafka-console-consumer.sh --zookeeper localhost:7181 >--consumer-timeout-ms 1000 --topic $topic -group fffffxx > > >the first try doesn't read anything out (of course) ; the second one >works. >but if I remove the first try, the remaining consumer can't read anything >either. > > > >looks like a bug? > >On Tue, Feb 24, 2015 at 1:51 PM, Yang <teddyyyy...@gmail.com> wrote: > >> in the following code, I just create a new topic, send some messages >>onto >> the new topic with console producer, then read it with console consumer. >> but after the 3rd line, nothing is printed out. >> >> >> >> export topic=ttnew2 >> sudo -u kafka bin/kafka-topics.sh --zookeeper localhost:7181 --create >> --topic $topic --partitions 1 --replication-factor 1 >> echo 'asdfasdfsadffadsfasf' | sudo -u kafka >> bin/kafka-console-producer.sh --broker-list localhost:9092 --topic >>$topic >> sudo -u kafka bin/kafka-console-consumer.sh --zookeeper localhost:7181 >> --topic $topic -group fffff >> >> >> shouldn't I expect the consumer on the 3rd line to read out the >>messages? >> in fact if I give it --from-beginning, it works fine >> >> >> >> this is kafka_2.9.2-0.8.1.1 >> >> >> Thanks >> Yang >>
