I think the issue is with " --from-latest" - this means consumers will
consume only data that arrives AFTER they start.
If you do that, first start consumers, leave them running, and then start
producers.
If you want to run producers first and only start consuming when producers
are done, remove this option.
Gwen
On Fri, Mar 20, 2015 at 1:41 PM, Saladi Naidu <naidusp2...@yahoo.com.invalid
> wrote:
> We are running 0.8.1.1, am experiencing following issue, any clue
> I have 7 topics in my Kafka cluster of 5 nodes. I pushed 1 million
> messages to each topic using perf producer tool (no compression, no
> acknowledgement) . I saw old messages about a bug in 0.8 Consumer code and
> it is fixed but I am still experiencing this issue
> /opt/kafka/kafka-0.8.1.1-src/bin/kafka-producer-perf-test.sh --topics
> ${dc}-perf-topic${topic} --messages 142857 --message-size 2000
> --broker-list ${listBroker} --threads ${threads} --request-num-acks ${acks}
> --compression-codec ${codec} > ${logFileProd}.${dc}.topic${topic}.log 2>&1
>
> After Producers populated topics, I run 7 consumers to consume from each
> above topic (1 consumer from 1 topic only) using Consumer Perf tool using
> below command
> /opt/kafka/kafka-0.8.1.1-src/bin/kafka-consumer-perf-test.sh --zookeeper
> ${listZoo}/kafka --from-latest --topic ${dc}-perf-topic${topic}
> --compression-codec ${codec} > ${logFileConsume}.${dc}.topic${topic}.log
> 2>&1
>
> All my consumers end up with following message
> start.time, end.time, fetch.size, data.consumed.in.MB, MB.sec,
> data.consumed.in.nMsg, nMsg.secSLF4J: Failed to load class
> "org.slf4j.impl.StaticLoggerBinder".SLF4J: Defaulting to no-operation (NOP)
> logger implementationSLF4J: See
> http://www.slf4j.org/codes.html#StaticLoggerBinder for further
> details.[2015-03-20 18:51:36,890] WARN
> [perf-consumer-68009_gckcnsys090001.c090.net-1426877494243-2472feec], No
> broker partitions consumed by consumer thread
> perf-consumer-68009_gckcnsys090001.1426877494243-2472feec-8 for topic
> dc7-perf-topic7 (kafka.consumer.ZookeeperConsumerConnector)[2015-03-20
> 18:51:36,903] WARN
> [perf-consumer-68009_gckcnsys090001.c090.net-1426877494243-2472feec], No
> broker partitions consumed by consumer thread
> perf-consumer-68009_gckcnsys090001.c090.net-1426877494243-2472feec-7 for
> topic dc7-perf-topic7
> (kafka.consumer.ZookeeperConsumerConnector)[2015-03-20 18:51:36,904] WARN
> [perf-consumer-68009_gckcnsys090001.c090.net-1426877494243-2472feec], No
> broker partitions consumed by consumer thread
> perf-consumer-68009_gckcnsys090001.c090.net-1426877494243-2472feec-4 for
> topic dc7-perf-topic7
> (kafka.consumer.ZookeeperConsumerConnector)[2015-03-20 18:51:36,906] WARN
> [perf-consumer-68009_gckcnsys090001.c090.net-1426877494243-2472feec], No
> broker partitions consumed by consumer thread
> perf-consumer-68009_gckcnsys090001.c090.net-1426877494243-2472feec-1 for
> topic dc7-perf-topic7
> (kafka.consumer.ZookeeperConsumerConnector)[2015-03-20 18:51:36,906] WARN
> [perf-consumer-68009_gckcnsys090001.c090.net-1426877494243-2472feec], No
> broker partitions consumed by consumer thread
> perf-consumer-68009_gckcnsys090001.c090.net-1426877494243-2472feec-5 for
> topic dc7-perf-topic7
> (kafka.consumer.ZookeeperConsumerConnector)[2015-03-20 18:51:36,907] WARN
> [perf-consumer-68009_gckcnsys090001.c090.net-1426877494243-2472feec], No
> broker partitions consumed by consumer thread
> perf-consumer-68009_gckcnsys090001.c090.net-1426877494243-2472feec-2 for
> topic dc7-perf-topic7
> (kafka.consumer.ZookeeperConsumerConnector)[2015-03-20 18:51:36,907] WARN
> [perf-consumer-68009_gckcnsys090001.c090.net-1426877494243-2472feec], No
> broker partitions consumed by consumer thread
> perf-consumer-68009_gckcnsys090001.c090.net-1426877494243-2472feec-9 for
> topic dc7-perf-topic7
> (kafka.consumer.ZookeeperConsumerConnector)[2015-03-20 18:51:36,907] WARN
> [perf-consumer-68009_gckcnsys090001.c090.net-1426877494243-2472feec], No
> broker partitions consumed by consumer thread
> perf-consumer-68009_gckcnsys090001.c090.net-1426877494243-2472feec-3 for
> topic dc7-perf-topic7
> (kafka.consumer.ZookeeperConsumerConnector)[2015-03-20 18:51:36,907] WARN
> [perf-consumer-68009_gckcnsys090001.c090.net-1426877494243-2472feec], No
> broker partitions consumed by consumer thread
> perf-consumer-68009_gckcnsys090001.c090.net-1426877494243-2472feec-6 for
> topic dc7-perf-topic7 (kafka.consumer.ZookeeperConsumerConnector)2015-03-20
> 18:51:37:939, 2015-03-20 18:51:42:960, 1048576, 0.0000, 0.0000, 0, 0.0000
>
> Naidu Saladi
>
