JiangJie: thanks for the info.
it looks I can change the default behavior and let a new group read from earliest offset by setting auto.offset.reset=smallest Yang On Tue, Feb 24, 2015 at 3:06 PM, Jiangjie Qin <j...@linkedin.com.invalid> wrote: > If a consumer comes from a new consumer group, it will by default consume > from the log end. > In you first scenario, because you started consumer after producing data, > when the consumer starts it will consume from the latest. So you will not > be able to see the messages even though they are in the broker. > In your second scenario, because consumer will commit the offsets > periodically and during rebalance. The first run of consumer will commit > offset to be 0. > Then you produced some data and run the second consumer. Because the > second run of consumer uses the same consumer group, it will pick up the > committed offset and consume from 0. In this case you will see the > produced messages. If you use a different consumer group in your second > run of consumer, you will not be able to see the messages because the > offset will be reset to latest like in the first scenario. > > -Jiangjie Qin > > On 2/24/15, 1:56 PM, "Yang" <teddyyyy...@gmail.com> wrote: > > >somehow I have to let the consumer run once BEFORE the messages are > >produced. then run consumer after messages are produced, then it works: > > > > > >export topic=ttnew21 > > sudo -u kafka bin/kafka-topics.sh --zookeeper localhost:7181 --create > >--topic $topic --partitions 1 --replication-factor 1 > >echo "first try" > > sudo -u kafka bin/kafka-console-consumer.sh --zookeeper localhost:7181 > >--consumer-timeout-ms 1000 --topic $topic -group fffffxx > > echo 'asdfasdfsadffadsfasf' | sudo -u kafka > >bin/kafka-console-producer.sh > >--broker-list localhost:9092 --topic $topic > >echo "second try" > > sudo -u kafka bin/kafka-console-consumer.sh --zookeeper localhost:7181 > >--consumer-timeout-ms 1000 --topic $topic -group fffffxx > > > > > >the first try doesn't read anything out (of course) ; the second one > >works. > >but if I remove the first try, the remaining consumer can't read anything > >either. > > > > > > > >looks like a bug? > > > >On Tue, Feb 24, 2015 at 1:51 PM, Yang <teddyyyy...@gmail.com> wrote: > > > >> in the following code, I just create a new topic, send some messages > >>onto > >> the new topic with console producer, then read it with console consumer. > >> but after the 3rd line, nothing is printed out. > >> > >> > >> > >> export topic=ttnew2 > >> sudo -u kafka bin/kafka-topics.sh --zookeeper localhost:7181 --create > >> --topic $topic --partitions 1 --replication-factor 1 > >> echo 'asdfasdfsadffadsfasf' | sudo -u kafka > >> bin/kafka-console-producer.sh --broker-list localhost:9092 --topic > >>$topic > >> sudo -u kafka bin/kafka-console-consumer.sh --zookeeper localhost:7181 > >> --topic $topic -group fffff > >> > >> > >> shouldn't I expect the consumer on the 3rd line to read out the > >>messages? > >> in fact if I give it --from-beginning, it works fine > >> > >> > >> > >> this is kafka_2.9.2-0.8.1.1 > >> > >> > >> Thanks > >> Yang > >> > >
