Hi Radu, Could you clarify what you mean by "I've noticed that if the kafka server was started a long time ago the console producer publishes all lines"? It's a known issue that if there are no brokers available, the producer will drop messages (https://issues.apache.org/jira/browse/KAFKA-1955) and if you've scripted this scenario, it's likely your broker hasn't come up completely before you start producing messages to it.
Regards, On Thu, Jun 16, 2016 at 3:19 AM, Radu Radutiu <rradu...@gmail.com> wrote: > Hi, > > I was following the Quickstart guide and I have noticed that > ConsoleProducer does not publish all messages (the number of messages > published differs from one run to another) and happens mostly on a fresh > started broker. > version: kafka_2.11-0.10.0.0 > OS: Linux (Ubuntu 14.04, Centos 7.2) > JDK: java version "1.7.0_101" > OpenJDK Runtime Environment (IcedTea 2.6.6) (7u101-2.6.6-0ubuntu0.14.04.1), > openjdk version "1.8.0_91" > OpenJDK Runtime Environment (build 1.8.0_91-b14) > > > How to reproduce: > - start zookeeper: > ~/work/kafka_2.11-0.10.0.0$ bin/zookeeper-server-start.sh > config/zookeeper.properties & > > -start kafka: > ~/work/kafka_2.11-0.10.0.0$ bin/kafka-server-start.sh > config/server.properties & > > -start console consumer (topic test1 is already created): > ~/work/kafka_2.11-0.10.0.0$ bin/kafka-console-consumer.sh > --bootstrap-server localhost:9092 -topic test1 --zookeeper localhost:2181 > > -in another terminal start console producer with the LICENSE file in kafka > directory as input: > ~/work/kafka_2.11-0.10.0.0$ bin/kafka-console-producer.sh --topic test1 > --broker-list localhost:9092 <LICENSE > > The last line in the console consumer output is not the last line in the > LICENSE file for the first few runs of the console producer. If I use the > --old-producer parameter, all the lines in the LICENSE file are published > (and appear in the console consumer output). Different runs of console > producer with the same input file publish different number of lines > (sometimes all, sometimes only 182 lines out of 330). I've noticed that if > the kafka server was started a long time ago the console producer publishes > all lines. > I have checked the kafka binary log file (in my case > /tmp/kafka-logs/test1-0/00000000000000000000.log ) and confirmed that the > messages are not published (the console consumer receives all the > messages). > > Is there an explanation for this behavior? > > Best regards, > Radu > -- Dustin Cote confluent.io
