Re: Difference between NOT_ENOUGH_REPLICAS and NOT_ENOUGH_REPLICAS_AFTER_APPEND

Thank you Gwen for the reply. Is there a way that I can prevent data duplication upon receiving NOT_ENOUGH_REPLICAS_AFTER_APPEND error? On Tue, 9 Jun 2015 at 00:27 Gwen Shapira wrote: > Hi, > > What you said is exactly correct :) > > We check ISR size twice. Once before writing to leader, and on

Kafka Server JMX Log End Offset Not updating after repartioning

Hi, Have notice an issue.  We retrieve the "kafka.log":type="Log",name="--LogEndOffset" 's Attribute: Value and used it to calculate the lag from consumer's offset. After we did a partition re-assignment, where partition leaders were changed, some of the new leader's  LogEndOffset were not updat

Re: High CPU usage for idle kafka server

I guess apache mailing list does not support attachments… Maybe you can paste it some where and send the link? Jiangjie (Becket) Qin From: "pundlik.anuja" mailto:pundlik.an...@gmail.com>> Reply-To: "users@kafka.apache.org" mailto:users@kafka.apache.org>> Date: Mond

Re: High CPU usage for idle kafka server

Hi Jiangjie, Resending profiling screenshot for High CPU usage on Idle kafka broker. Appreciate your help. Thanks, Anuja On Mon, Jun 8, 2015 at 1:45 PM, pundlik.anuja wrote: > Yes, partly it seems to be due to log cleaner thread. > > I changed log cleaning settings for each broker (server*.pr

what do 'soTimeout', 'bufferSize' and 'minBytes' mean for SimpleConsumer?

Hi, I'm using Kafka 0.8.2.1 SimpleConsumer . Could somebody clarify the meaning of a few config parameters for the SimpleConsumer and FetchREquestBuilder? My questions are below: -- Q1: in the signature of the SimpleConsumer constructor I see the Int 'soTimeout' parameter - what is the meanin

Re: High CPU usage for idle kafka server

Yes, partly it seems to be due to log cleaner thread. I changed log cleaning settings for each broker (server*.properties) file. # The minimum age of a log file to be eligible for deletion log.retention.hours=1 # A size-based retention policy for logs. Segments are pruned from the log as long a

Re: High CPU usage for idle kafka server

Could it also be that the log cleaner is running? This will definitely use some CPU while the cleaning is occurring (it would attempt to use one cpu per log cleaner thread you configure). -Jay On Mon, Jun 8, 2015 at 1:07 PM, Jiangjie Qin wrote: > It seems the attachments are lost. But high CPU

Re: High CPU usage for idle kafka server

It seems the attachments are lost. But high CPU for ReplicaRetcherThread might be related to KAFKA-1461. Can you try to apply that patch and see if it solves problem? Thanks, Jiangjie (Becket) Qin From: "pundlik.anuja" mailto:pundlik.an...@gmail.com>> Reply-To: "users@kafka.apache.org

Re: Anouncing PyKafka!

That's exciting news, looks very promising! I've added PyKafka to the Clients wiki page. Good work! 2015-06-08 21:24 GMT+02:00 Keith Bourgoin : > Hi Kafka folks, > > I'm happy to announce the 1.0 release of PyKafka > , a new Python driver for Kafka! > > This

Anouncing PyKafka!

Hi Kafka folks, I'm happy to announce the 1.0 release of PyKafka , a new Python driver for Kafka! This is an upgrade of an older library, named samsa, which was only compatible with Kafka 0.7.x. We've spent the last few months upgrading it to work with Kafka 0.

Re: High CPU usage for idle kafka server

Seems to be LogCleaner and FetchRequest threads. Attached is the profiling screenshot On Fri, Jun 5, 2015 at 3:06 PM, Jiangjie Qin wrote: > Has this to do with KAFKA-1461? > Can you see which thread is taking a lot of cpu? Some jconsole plugin can > get that information. > > Jiangjie (Becket) Qi

Re: callback handler is not getting called if cluster is down

KIP-19 should help in your case. Jiangjie (Becket) Qin On 6/8/15, 11:55 AM, "ankit tyagi" wrote: >Yes Jiangjie, > >I was using 1 broker with 1 replication factor for testing purpose. > >*Is there any way to detect broker failure with callback handler or any >other means while sending messages

Re: callback handler is not getting called if cluster is down

Yes Jiangjie, I was using 1 broker with 1 replication factor for testing purpose. *Is there any way to detect broker failure with callback handler or any other means while sending messages ??* On Mon, Jun 8, 2015 at 10:46 PM, Jiangjie Qin wrote: > What replication factor are you using? > > Cu

Re: New producer very slow to call Callback on error

KIP-19 should address this issue. https://cwiki.apache.org/confluence/display/KAFKA/KIP-19+-+Add+a+request+ti meout+to+NetworkClient Thanks, Jiangjie (Becket) Qin On 6/8/15, 10:44 AM, "Christofer Hedbrandh" wrote: >I think this question might relate to the very recently posted "callback >hand

New producer very slow to call Callback on error

I think this question might relate to the very recently posted "callback handler is not getting called if cluster is down" topic from "ankit tyagi". I am using the 0.8.2.1 new producer send(ProducerRecord record, Callback callback) with a Callback and never calling .get() on the Future. I have not

Re: simple consumer giving error always

That is offset out of range error. So you are fetching from an offset either greater than latest offset or smaller than earliest offset available on broker. Jiangjie (Becket) Qin On 6/8/15, 2:21 AM, "sunil kalva" wrote: >Hi >For few topics i always get FetchResponse.error code as "1", i am usin

Re: callback handler is not getting called if cluster is down

What replication factor are you using? Currently if a partition is offline, the message in producer will not be sent but sit in accumulator until the partition comes back online. Do you mean you want to use the message send callback to detect broker failure? Jiangjie (Becket) Qin On 6/8/15, 12:2

Re: Difference between NOT_ENOUGH_REPLICAS and NOT_ENOUGH_REPLICAS_AFTER_APPEND

Hi, What you said is exactly correct :) We check ISR size twice. Once before writing to leader, and once when checking for acks. The first error is thrown if we detect a small ISR before writing to the leader. The second if the ISR shrank after we wrote to the leader but before we got enough acks

Difference between NOT_ENOUGH_REPLICAS and NOT_ENOUGH_REPLICAS_AFTER_APPEND

Hi team, What is the difference between producer error NOT_ENOUGH_REPLICAS and NOT_ENOUGH_REPLICAS_AFTER_APPEND? Does the later one imply that the message has been written to the leader log successfully? If I have retry turned on in producer does it mean that duplicated messages may be written to

simple consumer giving error always

Hi For few topics i always get FetchResponse.error code as "1", i am using simple consumer 0.8.1. What are the reasons to get this error. and any fix for this ? -- SunilKalva

Re: Cascading failures on running out of disk space

Thanks Jason. We did run out of disk space and noticed IOExceptions too. No, the broker did not shut itself down. Is there some configuration that would enable this for one or all brokers? That would be a better scenario to be in. Right now, we have setup some alerts when disk space goes beyond a

callback handler is not getting called if cluster is down

Hi, we are using .8.2.0 broker and default async producer to send the message*. we recently found out that if whole cluster gets down then callback handler is not getting called while we are getting below exception continuously* *12:36:41,267# WARN [Selector] - Error in I/O with localhost/127.