Hi, Sorry for posting twice. It seems the issue I have encountered is indeed https://issues.apache.org/jira/browse/KAFKA-3129 as indicated by Vahid Hashemian in the other thread. If I change request.required.acks to 1 it works fine. As far as the problem going away if the broker is started a long time ago I cannot reproduce any longer. The problem occurs 50% of the time if request.required.acks=0 on my machine. If I use the --old-producer it works fine.
Best regards, Radu On 16 June 2016 at 16:15, Dustin Cote <dus...@confluent.io> wrote: > Hi Radu, > > Could you clarify what you mean by "I've noticed that if the kafka server > was started a long time ago the console producer publishes all lines"? > It's a known issue that if there are no brokers available, the producer > will drop messages (https://issues.apache.org/jira/browse/KAFKA-1955) and > if you've scripted this scenario, it's likely your broker hasn't come up > completely before you start producing messages to it. > > Regards, > > > On Thu, Jun 16, 2016 at 3:19 AM, Radu Radutiu <rradu...@gmail.com> wrote: > > > Hi, > > > > I was following the Quickstart guide and I have noticed that > > ConsoleProducer does not publish all messages (the number of messages > > published differs from one run to another) and happens mostly on a fresh > > started broker. > > version: kafka_2.11-0.10.0.0 > > OS: Linux (Ubuntu 14.04, Centos 7.2) > > JDK: java version "1.7.0_101" > > OpenJDK Runtime Environment (IcedTea 2.6.6) > (7u101-2.6.6-0ubuntu0.14.04.1), > > openjdk version "1.8.0_91" > > OpenJDK Runtime Environment (build 1.8.0_91-b14) > > > > > > How to reproduce: > > - start zookeeper: > > ~/work/kafka_2.11-0.10.0.0$ bin/zookeeper-server-start.sh > > config/zookeeper.properties & > > > > -start kafka: > > ~/work/kafka_2.11-0.10.0.0$ bin/kafka-server-start.sh > > config/server.properties & > > > > -start console consumer (topic test1 is already created): > > ~/work/kafka_2.11-0.10.0.0$ bin/kafka-console-consumer.sh > > --bootstrap-server localhost:9092 -topic test1 --zookeeper localhost:2181 > > > > -in another terminal start console producer with the LICENSE file in > kafka > > directory as input: > > ~/work/kafka_2.11-0.10.0.0$ bin/kafka-console-producer.sh --topic test1 > > --broker-list localhost:9092 <LICENSE > > > > The last line in the console consumer output is not the last line in the > > LICENSE file for the first few runs of the console producer. If I use the > > --old-producer parameter, all the lines in the LICENSE file are published > > (and appear in the console consumer output). Different runs of console > > producer with the same input file publish different number of lines > > (sometimes all, sometimes only 182 lines out of 330). I've noticed that > if > > the kafka server was started a long time ago the console producer > publishes > > all lines. > > I have checked the kafka binary log file (in my case > > /tmp/kafka-logs/test1-0/00000000000000000000.log ) and confirmed that the > > messages are not published (the console consumer receives all the > > messages). > > > > Is there an explanation for this behavior? > > > > Best regards, > > Radu > > > > > > -- > Dustin Cote > confluent.io >
