Broker don't get back in ISR when killed and restarted

Hi There, Zookeeper version: 3.4.6-1569965, built on 02/20/2014 09:09 GMT Kafka version: kafka_2.8.0-0.8.1.1 192.168.1.180 zookeeper:2181 broker:9092(broker.id=1) zookeeper.connect=localhost:2181 192.168.1.190 broker:9092(broker.id=2) zookeeper.connect=192.168.1.180:2181 1) I start both kafka

After creating a topic, broker gets dropped from ISR

Hi again, 192.168.1.180 Zk: 192.168.1.180:2181 Kafka: 9092 Broker.id = 1 zookeeper.connect=192.168.1.180:2181 -- 192.168.1.190 Zk: 192.168.1.180:2181 Kafka: 9092 Broker.id = 2 zookeeper.connect=192.168.1.180:2181 -- I start both Kafka-server I create a topic using the following command

Re: AWS EC2 deployment best practices

I didn't know about KAFKA-1215, thanks. I'm not sure it would fully address my concerns of a producer writing to the partition leader in different AZ, though. To answer your question, I was thinking ephemerals with replication, yes. With a reservation, it's pretty easy to get e.g. two i2.xlarge fo

Re: AWS EC2 deployment best practices

OK, yeah, speaking from experience I would be comfortable with using the ephemeral storage if it's replicated across AZs. More and more EC2 instances have local SSDs, so you'll get great IO. Of course, you better monitor your instance, and if a instance terminates, you're vulnerable if a second

RE: BadVersion state in Kafka Logs

I would love to help you guys to make Kafka best in Pub/Sub, will continue doing that whenever I can. Do we have 0.8.1.2 release tag or should we apply patch on top of 0.8.1.1 tag because we need this KAFKA-1382 JIRA ?. Balaji From: Neha Narkhede [mailto:neha.narkh...@gmail.com] Sent: Monday,

Re: multi-node and multi-broker kafka cluster setup

Hello, In general it is not required to have the kafka brokers installed on the same nodes of the zk servers, and each node can host multiple kafka brokers: you just need to make sure they do not share the same port and the same data dir. Guozhang On Mon, Sep 29, 2014 at 8:31 PM, Sa Li wrote:

Re: BadVersion state in Kafka Logs

Does the patch in KAFKA-1382 apply on the 0.8.1 branch? If not if you could make a patch that does would be great. I will kick off a discussion for KAFKA-1382 and the scala 2.11 for 0.8.1.2 release (and see what others may think we should do like the gradle changes I think we should do too for sr

Re: rebalance brokers

In 0.7 one is required to manually create partitions for existing topics when we add new brokers (i.e. modify the zk registration path). On Mon, Sep 29, 2014 at 9:07 PM, Guangle Fan wrote: > Guozhang, yes, the version running on there is still 0.7, you are right, > there is no such concept of re

kafka & docker

Hi, Any one has dockerized kafka working? Should we specify the ip address? I expected everything working on just localhost but saw this "SEVERE: Producer connection to 172.17.0.3:9092 unsuccessful". Thanks in advance! Best Regards, Mingtao

RE: BadVersion state in Kafka Logs

Hi Joe, I did not try on 0.8.1 branch ,I can try and see if it goes through when I get some breather. Thanks for initiating on 0.8.1.2. Thanks, Balaji -Original Message- From: Joe Stein [mailto:joe.st...@stealth.ly] Sent: Tuesday, September 30, 2014 9:34 AM To: users@kafka.apache.org

Re: kafka & docker

You need to change the advertised hostname. Take a look https://registry.hub.docker.com/u/stealthly/docker-kafka/ and https://registry.hub.docker.com/u/stealthly/docker-zookeeper/ we use it often for local testing here is how to start https://github.com/stealthly/docker-kafka/blob/master/start-bro

[DISCUSS] 0.8.1.2 Release

Hi, I wanted to kick off a specific discussion on a 0.8.1.2 release. Here are the JIRAs I would like to propose to back port a patch (if not already done so) and apply them to the 0.8.1 branch for a 0.8.1.2 release https://issues.apache.org/jira/browse/KAFKA-1502 (source jar is empty) https://iss

Still Stale TopicMetadata

Hi Kafka users, Was there ever a JIRA ticket filed for this? "Re: Stale TopicMetadata" http://mail-archives.apache.org/mod_mbox/kafka-users/201307.mbox/%3ce238b018f88c39429066fc8c4bfd0c2e019be...@esv4-mbx01.linkedin.biz%3E As far as I can tell this is still an issue in 0.8.1.1 Using the python

Re: Still Stale TopicMetadata

I believe this is the ticket https://issues.apache.org/jira/browse/KAFKA-972 /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop

Created topic by 2 partitions, only can use the one partition

Hi, Guys It is very weird, I created a topic with 2 partitions couple weeks ago, and I can only production the message to partition 0, not partition 1, but for now, I created a new topic again with 2 partitions, it does work. So whats problem of the old topic? I tried to describe the old topic, I f

Re: kafka & docker

Thanks Joe.. seems quite handy. Is there a 'Kafka->HDFS with Camus' docker as well one can play around with? On Tue, Sep 30, 2014 at 9:00 AM, Joe Stein wrote: > You need to change the advertised hostname. > > Take a look https://registry.hub.docker.com/u/stealthly/docker-kafka/ and > https://reg

Re: kafka & docker

Hi Joe What's the story for persisting data with Docker? Do you use a data volume or do you just start fresh every time you start the Docker instance? Daniel. > On 1/10/2014, at 7:13 am, Buntu Dev wrote: > > Thanks Joe.. seems quite handy. Is there a 'Kafka->HDFS with Camus' docker > as well

Re: kafka & docker

<< Is there a 'Kafka->HDFS with Camus' docker as well one can play around with? Not that I know of. These folks http://blog.sequenceiq.com/blog/2014/09/15/hadoop-2-5-1-docker/ have nice Hadoop docker containers may be a good starting point. << What's the story for persisting data with Docker? D

Re: [DISCUSS] 0.8.1.2 Release

Can we discuss the need for 0.8.1.2? I'm wondering if it's related to the timeline of 0.8.2 in any way? For instance, if we can get 0.8.2 out in the next 2-3 weeks, do we still need to get 0.8.1.2 out or can people just upgrade to 0.8.2? On Tue, Sep 30, 2014 at 9:53 AM, Joe Stein wrote: > Hi, I

RE: [DISCUSS] 0.8.1.2 Release

In DISH we are having issues in 0.8-beta version used in PROD, it's crashing every 2 days and becoming a blocker for us. It would be great if we get 0.8.2 or 0.8.1.2 whichever is faster as we can't wait for 3 weeks as our new Order Management system is going to sit on top of Kafka. -Origin

Re: [DISCUSS] 0.8.1.2 Release

I was one asking for 0.8.1.2 a few weeks back, when 0.8.2 was at least 6-8 weeks out. If we truly believe that 0.8.2 will go “golden” and stable in 2-3 weeks, I, for one, don’t need a 0.8.1.2, but it depends on the confidence in shipping 0.8.2 soonish. YMMV, -Jonathan On Sep 30, 2014, at 12

Re: BadVersion state in Kafka Logs

Have you tried increasing your broker's zookeeper session timeout as a work around for now to alleviate the issue? Is that an option for you? Assuming that is the culprit you are timing zk sessions out and bumping into KAFKA-1382 on the reconnect? Not knowing enough about what is going on with the

Re: multi-node and multi-broker kafka cluster setup

Hi Sa While it's possible to run multiple brokers on a single machine, I would be interested to hear why you would want to. Kafka is very efficient and can use all of the system resources under load. Running multiple brokers would increase zookeeper load, force resource sharing between the Kafk

RE: BadVersion state in Kafka Logs

The zookeeper session timeout is 60 secs ,but that did not help. We are having broker crash and unresponsive, we got the "conditional update" failed error when broker crashed which confirmed that it is because of KAFKA-1382. server.log.2014-09-23:2014-09-23 13:54:48 ERROR utils.ZkUtils$ - Condi

Re: BadVersion state in Kafka Logs

It sounds like you have a much deeper rooted problem. Is zookeeper swapping? Something has to be causing this. After you fix this symptom you will probably start to see constant leader elections and the isr shrinking/growing and constant consumer rebalancing (or at least every minute) and a herd

Re: Zookeeper reconnect failed due to 'state changed (Expired)'

With ack=1, acked messages could be lost when the leader fails. Thanks, Jun On Mon, Sep 29, 2014 at 8:04 AM, Andrew Otto wrote: > This happened again to me this weekend. I've done some sleuthing, and > definitely can see some crazy paging stats when this lock up happens. For > the curious, m

Re: Created topic by 2 partitions, only can use the one partition

Hi Jiang, Which producer client did you use? And did you specify any keys for your sent messages? Guozhang On Tue, Sep 30, 2014 at 10:45 AM, Jiang Jacky wrote: > Hi, Guys > It is very weird, I created a topic with 2 partitions couple weeks ago, and > I can only production the message to partit

Re: BadVersion state in Kafka Logs

Also check for really long/bad GC pauses as another possibility. Not sure your JDK and JVM_OPTS and if you are setting like this https://kafka.apache.org/documentation.html#java or not. You need to find some "spike" somewhere right before that error happens to track down what is causing the timeout

Re: AWS EC2 deployment best practices

I'm also interested in hearing more about deploying Kafka in AWS. I was also considering options like your 1a and 2. I ran some calculations and one interesting thing I ran across was bandwidth costs between AZs. In 1a, if you can have your producers and consumers in the same AZ as the "master"

Re: After creating a topic, broker gets dropped from ISR

Could you check that the broker host registered in ZK is the ip that you are expecting? Thanks, Jun On Tue, Sep 30, 2014 at 3:03 AM, florent valdelievre < florentvaldelie...@gmail.com> wrote: > Hi again, > > 192.168.1.180 > Zk: 192.168.1.180:2181 > Kafka: 9092 > Broker.id = 1 > zookeeper.connec

Re: Still Stale TopicMetadata

Actually, this is probably the more relevant jira: KAFKA-1367. Thanks, Jun On Tue, Sep 30, 2014 at 10:00 AM, Christofer Hedbrandh < christo...@knewton.com> wrote: > Hi Kafka users, > > Was there ever a JIRA ticket filed for this? > > "Re: Stale TopicMetadata" > > > http://mail-archives.apache.o