Watch this talk. Kafka will not lose messages it configured correctly.

http://www.confluent.io/kafka-summit-2016-ops-when-it-absolutely-positively-has-to-be-there
 
<http://www.confluent.io/kafka-summit-2016-ops-when-it-absolutely-positively-has-to-be-there>

-hans





> On Oct 13, 2016, at 2:48 AM, <yangy...@sina.com> <yangy...@sina.com> wrote:
> 
> Other question:   if account of consumer over broker number, whether loss 
> data?    if some broker rebooted, whether loss data? 
> Thanks.
> 
> --------------------------------
> 
> 
> ----- 原始邮件 -----
> 发件人:Guozhang Wang <wangg...@gmail.com>
> 收件人:yangy...@sina.com
> 抄送人:users <users@kafka.apache.org>
> 主题:Re: Re: I found kafka lsot message
> 日期:2016年10月12日 07点05分
> 
> One common issue of lost messages is that consumer auto-committing (related 
> config is "auto.commit.enabled", "commit.interval.ms"): from the Kafka 
> consumer point of view, once the messages are returned from the "poll" call 
> they are considered "consumed", and if committing offsets is called it will 
> set the offset to the last message's offset. So if you have some exceptions / 
> errors in the middle of processing some of your consumed records, upon 
> restarting they will not be re-fetched again, hence possibly causing "data 
> loss" on your end.
> 
> Guozhang
> On Mon, Oct 10, 2016 at 3:33 AM,  <yangy...@sina.com> wrote:
> Hi Guozhang,
> At first, thank you answer my question, and give me some suggest.But, I'm 
> sure I readed some introduction about kafka.
> In my producer, My Code is( c code):res = rd_kafka_conf_set(kafka_conf, 
> "request.required.acks", "-1", NULL, 0);res = rd_kafka_topic_conf_set( 
> topic_conf, "produce.offset.report", "true", errstr, sizeof(errstr) );
> In my consumer, My Code is(kafka-python):
>                self.__consumer = KafkaConsumer( 
> bootstrap_servers=self.__kafka_config["hosts"],                               
>              group_id=self.__kafka_config["group"],                           
>                  auto_offset_reset="earliest",                                
>             )                
> self.__consumer.subscribe([self.__kafka_config["data_topic"]])
>                for message in self.__consumer:
> 
> I think these codes is common, What's your suggest about these codes?
> In the end, I must explain: Lost message is not often, some time, couple days 
> can find one or two lost messages.But some day, maybe can find over 20 
> messages were losted.Our message over 1,200,000 in one day.  
> So, do your have any suggest?Bye the way, can you speak Chinese? Thank you 
> very much & Best wishesJerry
> 
> --------------------------------
> 
> 
> ----- 原始邮件 -----
> 发件人:Guozhang Wang <wangg...@gmail.com>
> 收件人:"users@kafka.apache.org" <users@kafka.apache.org>, yangy...@sina.com
> 主题:Re: I found kafka lsot message
> 日期:2016年10月10日 01点25分
> 
> Jerry,
> Message lost scenarios usually are due to producer and consumer 
> mis-configured. Have you read about the client configs web docs?
> http://kafka.apache.org/documentation#producerconfigs
> 
> http://kafka.apache.org/documentation#newconsumerconfigs
> 
> 
> If not I'd suggest you reading those first and see if you can tune some of 
> these configs to have better delivery guarantees.
> Guozhang
> 
> On Fri, Oct 7, 2016 at 9:48 PM,  <yangy...@sina.com> wrote:
> Hello every body,I build a kafka cluster(include 5 domains) use 
> kafka_2.11-0.10.0.0 and kafka-python-1.3.1.I create a topic by 100 partitions 
> and 2 replicate, then use 20 consumers to receive message.
> 
> But, I found sometime the kafka lost message! I found some partition's 
> offsite lost at consumer.After, I make a global index for every message from 
> producer for confirm this problem, and I also found the global index had been 
> break!
> 
> Why the kafka lost message?  What can I do to fix the problem?
> 
> Thanks!Jerry
> 
> 
> 
> --------------------------------
> 
> 
> 
> 
> 
> -- 
> -- Guozhang
> 
> 
> 
> 
> 
> -- 
> -- Guozhang
> 
> 
> 

Reply via email to