Re: Auto preferred leader elections cause data loss?

Hi Gwen, Thanks for reply. Just realized *replica.lag.time.max.ms and replica.lag.max.messages *just work when leader checking maybe shrink ISR. So for this scenario it's safe. Then I think what I've been through is caused by follower's HW always <= leader's, just

Re: port already in use error when trying to add topic

This is not a bug. The java process spawned by kafka-topics.sh is trying to bind to 9998 upon start. The java process spawned by kafka-server-start.sh already owns that port. It's doing this because both of these scripts use kafka-run-class.sh and that is where you defined your 'export JMX_PORT'.

Re: port already in use error when trying to add topic

After completely disabling JMX settings, i was able to create topics. Seems like there is an issue with using JMX with the product. Should i create bug? On Sun, Sep 13, 2015 at 9:07 PM, allen chan wrote: > Changing the port to 9998 did not help. Still the same error occurred > > On Sat, Sep 12,

Re: 0.9.0.0 remaining jiras

Is KAFKA-1811 worth considering for 0.9.0? Thanks! Eddie On Mon, Sep 14, 2015 at 4:34 PM, Jiangjie Qin wrote: > HI Jun, > > Can we also include KAFKA-2448 in 0.9 as well? We see this issue a few > times before and that cause replica fetcher th

Thread safety when committing from multiple threads

Hi, I have a multi-threaded Kafka consumer. I have auto commit disabled and from each thread, I periodically call commitOffsets() to commit the offsets to Kafka. Is this thread safe ? What is the effect of calling commitOffsets() concurrently from multiple threads ? Should the consumer writer ens

Re: producer api

Thank you Erik. But in my setup, there is only one node whose public ip provided in my broker cluster, so I can only use one bootstrap broker as for now. On Mon, Sep 14, 2015 at 3:50 PM, Helleren, Erik wrote: > You only need one of the brokers to connect for publishing. Kafka will > tell the c

Re: producer api

You only need one of the brokers to connect for publishing. Kafka will tell the client about all the other brokers. But best practices state including all of them is best. -Erik On 9/14/15, 2:46 PM, "Yuheng Du" wrote: >I am writing a kafka producer application in java. I want the producer to >

producer api

I am writing a kafka producer application in java. I want the producer to publish data to a cluster of 6 brokers. Is there a way to specify only the load balancing node but not all the brokers list? For example, like in the benchmarking kafka commandssdg: bin/kafka-run-class.sh org.apache.kafka.c

Re: 0.9.0.0 remaining jiras

HI Jun, Can we also include KAFKA-2448 in 0.9 as well? We see this issue a few times before and that cause replica fetcher threads not startup. Thanks, Jiangjie (Becket) Qin On Sat, Sep 12, 2015 at 9:40 AM, Jun Rao wrote: > The following is a candidate list of jiras that we want to complete i

Unreasonably high CPU from Kafka (0.8.2.1)

We have been using Kafka for a while now in one of dev projects. Currently we have just 1 broker and 1 zookeeper instance. Almost every day, Kafka "stalls" and we end up cleaning up the data/log folder of Kafka and zookeeper and bring it up afresh. We haven't been able to narrow down the issue

Re: Unclean leader election docs outdated

Yes you are right. Could you file a JIRA to edit the documents? Guozhang On Fri, Sep 11, 2015 at 4:41 PM, Stevo Slavić wrote: > That sentence is in both > https://svn.apache.org/repos/asf/kafka/site/083/design.html and > https://svn.apache.org/repos/asf/kafka/site/082/design.html near the end >

Re: Broker restart, new producer and snappy in 0.8.2.1

Ok Thanks Joe, Will try 0.8.2.2 producer Vidhya On Sun, Sep 13, 2015 at 7:12 AM, Joe Stein wrote: > Hi, the 0.8.2.2 release (which vote just passed and should be announced > soon) has a patch that may be related > https://issues.apache.org/jira/browse/KAFKA-2308 not sure. > > Here are the 0.8.2

Re: Tools/recommendations to debug performance issues?

Kafka also collects very useful metrics on request times and their breakdown. They are under kafka.network. On Mon, Sep 14, 2015 at 6:59 AM, Rahul Jain wrote: > Have you checked the consumer lag? You can use the offset checker tool to > see if there is a lag. > On 14 Sep 2015 18:36, "noah" wr

Re: Auto preferred leader elections cause data loss?

acks = all should prevent this scenario: If broker 0 is still in ISR, the produce request for 101 will not be "acked" (because 0 is in ISR and not available for acking), and the producer will retry it until all ISR acks. If broker 0 dropped off ISR, it will not be able to rejoin until it has all

Re: 0.9.0.0 remaining jiras

Agree that these are very nice to have. We've seen many deployments that need to manage these on their own. However, if this is not ready before we are done adding security and the new consumer, it will make sense to still release 0.9.0 and add the broker management improvements in 0.9.1. I'm tryi

Re: 0.9.0.0 remaining jiras

We decided to rename 0.8.3 to 0.9.0 since it contains few large changes (Security, new consumer, quotas). On Sun, Sep 13, 2015 at 11:56 PM, Jason Rosenberg wrote: > Hi Jun, > > Can you clarify, will there not be a 0.8.3.0 (and instead we move straight > to 0.9.0.0)? > > Also, can you outline t

Re: 0.9.0.0 remaining jiras

Jun, Would be nice to have https://issues.apache.org/jira/browse/KAFKA-2106 (if not that, than related https://issues.apache.org/jira/browse/KAFKA-1792 ) in 0.9 release. Both have patch provided. If KAFKA-2106 is delivered, maybe KAFKA-1792 is redundant, not needed. For some reason KAFKA-2106 has

Re: Auto preferred leader elections cause data loss?

Hi group, I think I've hit the KAFKA-1561 and KAFKA-1211, that is follower's HW is always <= leader's HW, then when leader transits, data loss can happen... So what we could do is try to do every thing to avoid leader transition, right? 2015-09-14 11:59 GMT+08:00 Zhao Weinan : > Hi group, > > S

Re: What can be reason for fetcher thread for slow response.

Madhukar, To me, the broker config looks good. The issue I see is that there is large number of synchronous producers, spamming the kafka brokers with a lot of singular appends. I think the suggested approach now is to use the new producer API: http://kafka.apache.org/082/javadoc/org/apache/kafk

Re: Tools/recommendations to debug performance issues?

Have you checked the consumer lag? You can use the offset checker tool to see if there is a lag. On 14 Sep 2015 18:36, "noah" wrote: > We're using 0.8.2.1 processing maybe 1 million messages per hour. Each > message includes tracking information with a timestamp for when it was > produced, and a

Tools/recommendations to debug performance issues?

We're using 0.8.2.1 processing maybe 1 million messages per hour. Each message includes tracking information with a timestamp for when it was produced, and a timestamp for when it was consumed, to give us roughly the amount of time it spent in Kafka. On average this number is in the seconds and ou

problems about setting up environment

Hi I`m try to learning kafka. so I read wiki and follow that (https://cwiki.apache.org/confluence/display/KAFKA/Developer+Setup) and I finally import projects on eclipse workspace. But here are some problems about compile errors. 1. In 'examples' project : 1) kafka.examples.Consumer.java K

Re: 0.9.0.0 remaining jiras

Hello Jason, Maybe this answers your question: http://mail-archives.apache.org/mod_mbox/kafka-dev/201509.mbox/%3CCAFc58G-UScVKrSF1kdsowQ8Y96OAaZEdiZsk40G8fwf7iToFaw%40mail.gmail.com%3E Kind regards, Stevo Slavic. On Mon, Sep 14, 2015 at 8:56 AM, Jason Rosenberg wrote: > Hi Jun, > > Can you cl