Watch this talk. Kafka will not lose messages it configured correctly. http://www.confluent.io/kafka-summit-2016-ops-when-it-absolutely-positively-has-to-be-there <http://www.confluent.io/kafka-summit-2016-ops-when-it-absolutely-positively-has-to-be-there>
-hans > On Oct 13, 2016, at 2:48 AM, <yangy...@sina.com> <yangy...@sina.com> wrote: > > Other question: if account of consumer over broker number, whether loss > data? if some broker rebooted, whether loss data? > Thanks. > > -------------------------------- > > > ----- 原始邮件 ----- > 发件人：Guozhang Wang <wangg...@gmail.com> > 收件人：yangy...@sina.com > 抄送人：users <users@kafka.apache.org> > 主题：Re: Re: I found kafka lsot message > 日期：2016年10月12日 07点05分 > > One common issue of lost messages is that consumer auto-committing (related > config is "auto.commit.enabled", "commit.interval.ms"): from the Kafka > consumer point of view, once the messages are returned from the "poll" call > they are considered "consumed", and if committing offsets is called it will > set the offset to the last message's offset. So if you have some exceptions / > errors in the middle of processing some of your consumed records, upon > restarting they will not be re-fetched again, hence possibly causing "data > loss" on your end. > > Guozhang > On Mon, Oct 10, 2016 at 3:33 AM, <yangy...@sina.com> wrote: > Hi Guozhang, > At first, thank you answer my question, and give me some suggest.But, I'm > sure I readed some introduction about kafka. > In my producer, My Code is( c code):res = rd_kafka_conf_set(kafka_conf, > "request.required.acks", "-1", NULL, 0);res = rd_kafka_topic_conf_set( > topic_conf, "produce.offset.report", "true", errstr, sizeof(errstr) ); > In my consumer, My Code is(kafka-python): > self.__consumer = KafkaConsumer( > bootstrap_servers=self.__kafka_config["hosts"], > group_id=self.__kafka_config["group"], > auto_offset_reset="earliest", > ) > self.__consumer.subscribe([self.__kafka_config["data_topic"]]) > for message in self.__consumer: > > I think these codes is common, What's your suggest about these codes? > In the end, I must explain: Lost message is not often, some time, couple days > can find one or two lost messages.But some day, maybe can find over 20 > messages were losted.Our message over 1,200,000 in one day. > So, do your have any suggest?Bye the way, can you speak Chinese? Thank you > very much & Best wishesJerry > > -------------------------------- > > > ----- 原始邮件 ----- > 发件人：Guozhang Wang <wangg...@gmail.com> > 收件人："users@kafka.apache.org" <users@kafka.apache.org>, yangy...@sina.com > 主题：Re: I found kafka lsot message > 日期：2016年10月10日 01点25分 > > Jerry, > Message lost scenarios usually are due to producer and consumer > mis-configured. Have you read about the client configs web docs? > http://kafka.apache.org/documentation#producerconfigs > > http://kafka.apache.org/documentation#newconsumerconfigs > > > If not I'd suggest you reading those first and see if you can tune some of > these configs to have better delivery guarantees. > Guozhang > > On Fri, Oct 7, 2016 at 9:48 PM, <yangy...@sina.com> wrote: > Hello every body,I build a kafka cluster(include 5 domains) use > kafka_2.11-0.10.0.0 and kafka-python-1.3.1.I create a topic by 100 partitions > and 2 replicate, then use 20 consumers to receive message. > > But, I found sometime the kafka lost message! I found some partition's > offsite lost at consumer.After, I make a global index for every message from > producer for confirm this problem, and I also found the global index had been > break! > > Why the kafka lost message? What can I do to fix the problem? > > Thanks!Jerry > > > > -------------------------------- > > > > > > -- > -- Guozhang > > > > > > -- > -- Guozhang > > >
