I believe this issue is similar to the one reported here: https://issues.apache.org/jira/browse/KAFKA-3129.
--Vahid From: Dean Arnold <renodino...@gmail.com> To: users@kafka.apache.org Date: 06/15/2016 11:27 AM Subject: Re: ConsoleProducer missing messages (random behavior) btw, it appears the missing msgs are at the end of the CSV file, so maybe the producer doesn't properly flush when it gets EOF on stdin ? On Wed, Jun 15, 2016 at 11:21 AM, Dean Arnold <renodino...@gmail.com> wrote: > I'm seeing similar issues with 0.9.0.1. > > I'm feeding CSV records (65536 total, 1 record per msg) to the console > producer, which are consumed via a sink connector (using connect-standalone > and a single partition). The sink occasionally reports flushing less than > 65536 msgs via the sink flush(). Restarting the sink connector with a > forced reset to offset 0 (ie, replaying all the msgs on the topic) shows > that the messages are still missing (ie, no gaps in offsets), so I assume > the msgs must be lost by the producer ? > > > On Wed, Jun 15, 2016 at 1:29 AM, Radu Radutiu <rradu...@gmail.com> wrote: > >> Hi, >> >> I was following the Quickstart guide and I have noticed that >> ConsoleProducer does not publish all messages (the number of messages >> published differs from one run to another) and happens mostly on a fresh >> started broker. >> version: kafka_2.11-0.10.0.0 >> OS: Linux (Ubuntu 14.04, Centos 7.2) >> JDK: java version "1.7.0_101" >> OpenJDK Runtime Environment (IcedTea 2.6.6) >> (7u101-2.6.6-0ubuntu0.14.04.1), >> openjdk version "1.8.0_91" >> OpenJDK Runtime Environment (build 1.8.0_91-b14) >> >> >> How to reproduce: >> - start zookeeper: >> ~/work/kafka_2.11-0.10.0.0$ bin/zookeeper-server-start.sh >> config/zookeeper.properties & >> >> -start kafka: >> ~/work/kafka_2.11-0.10.0.0$ bin/kafka-server-start.sh >> config/server.properties & >> >> -start console consumer (topic test1 is already created): >> ~/work/kafka_2.11-0.10.0.0$ bin/kafka-console-consumer.sh >> --bootstrap-server localhost:9092 -topic test1 --zookeeper localhost:2181 >> >> -in another terminal start console producer with the LICENSE file in kafka >> directory as input: >> ~/work/kafka_2.11-0.10.0.0$ bin/kafka-console-producer.sh --topic test1 >> --broker-list localhost:9092 <LICENSE >> >> The last line in the console consumer output is not the last line in the >> LICENSE file for the first few runs of the console producer. If I use the >> --old-producer parameter, all the lines in the LICENSE file are published >> (and appear in the console consumer output). Different runs of console >> producer with the same input file publish different number of lines >> (sometimes all, sometimes only 182 lines out of 330). I've noticed that if >> the kafka server was started a long time ago the console producer >> publishes >> all lines. >> I have checked the kafka binary log file (in my case >> /tmp/kafka-logs/test1-0/00000000000000000000.log ) and confirmed that the >> messages are not published (the console consumer receives all the >> messages). >> >> Is there an explanation for this behavior? >> >> Best regards, >> Radu >> > >
