How to monitor consumer connection？

Hi, In kafka 0.7.2, we use 2 brokers and 12 consumers. I have 2 questions: 1. seems if one broker is restarted then all consumers need restart to reconnect to kafka. right? 2. How to monitor whether the consumer connection with the broker is healthy? Thank you very much.

Re: partition reassignment stuck on 0.8

On Thu, Dec 12, 2013 at 9:46 PM, Jun Rao wrote: > Since we don't support delete topics yet, you would have to wipe out all ZK > and kafka logs. > > Thanks, > > Jun > > got it and done. so it sounds like i should run a number of disparate clusters to spread risk for topics since a partition is an

Re: partition reassignment stuck on 0.8

Since we don't support delete topics yet, you would have to wipe out all ZK and kafka logs. Thanks, Jun On Thu, Dec 12, 2013 at 9:32 PM, David Birdsong wrote: > On Thu, Dec 12, 2013 at 9:28 PM, Jun Rao wrote: > > > Could you try starting from scratch again? The recent fix that we had may > >

Re: partition reassignment stuck on 0.8

On Thu, Dec 12, 2013 at 9:28 PM, Jun Rao wrote: > Could you try starting from scratch again? The recent fix that we had may > not be able to recover a cluster already in an inconsistent state. > > Thanks, > > Jun > > > How does one start from scratch? Wipe ZK, is there some state file? I have oth

Re: partition reassignment stuck on 0.8

On Thu, Dec 12, 2013 at 9:28 PM, Guozhang Wang wrote: > David, > > Could you try to see if this is due to > https://issues.apache.org/jira/browse/KAFKA-1178? > > Guozhang > Which node do I look for this on? Leader? ISR-candidate? Controller? > > > On Thu, Dec 12, 2013 at 8:45 PM, David Birdson

Re: partition reassignment stuck on 0.8

David, Could you try to see if this is due to https://issues.apache.org/jira/browse/KAFKA-1178? Guozhang On Thu, Dec 12, 2013 at 8:45 PM, David Birdsong wrote: > I was running a 2-node kafka cluster off github trunnk at: > eedbea6526986783257ad0e025c451a8ee3d9095 > > ...for a few weeks with no

Re: partition reassignment stuck on 0.8

Could you try starting from scratch again? The recent fix that we had may not be able to recover a cluster already in an inconsistent state. Thanks, Jun On Thu, Dec 12, 2013 at 8:45 PM, David Birdsong wrote: > I was running a 2-node kafka cluster off github trunnk at: > eedbea6526986783257ad0e

Re: Consumer not able to create message streams

When debugging, be careful not to print the stream itself since it will block forever. Thanks, Jun On Thu, Dec 12, 2013 at 12:41 PM, Tarang Dawer wrote: > While debugging, the code got stuck on this call. tried the same by placing > logs after each call, noticed that whenever consumer got stuc

Re: Scala 2.10.1 - Issues with starting zookeeper

using ./sbt "++ 2.10.1 assembly-package-dependency" Thanks, Jun On Thu, Dec 12, 2013 at 9:19 PM, Jun Rao wrote: > Try building assembly dependency again after building the 2.10.1 package. > > Thanks, > > Jun > > > On Thu, Dec 12, 2013 at 12:35 PM, Sridhar Sreenivasan < > sridharsre...@gmail.c

Re: Scala 2.10.1 - Issues with starting zookeeper

Try building assembly dependency again after building the 2.10.1 package. Thanks, Jun On Thu, Dec 12, 2013 at 12:35 PM, Sridhar Sreenivasan < sridharsre...@gmail.com> wrote: > Hi, > I have downloaded > kafka-0.8.0-src.tgz< > https://www.apache.org/dyn/closer.cgi?path=/kafka/0.8.0/kafka-0.8

Re: Topic not created

Yes, could you file a jira to track that? Thanks, Jun On Thu, Dec 12, 2013 at 8:58 AM, Barto Rael wrote: > Thanks Jun for response. > > So the problem i am facing is expected behavior of kafka. > > I have just a suggestion that in case of live brokers less than # replicas: > There should be t

partition reassignment stuck on 0.8

I was running a 2-node kafka cluster off github trunnk at: eedbea6526986783257ad0e025c451a8ee3d9095 ...for a few weeks with no issues. I recently downloaded the 0.8 stable version, configured/started two new brokers with 0.8. I successfully reassigned all but 1 partition from the older pair to th

Re: Consumer not able to create message streams

While debugging, the code got stuck on this call. tried the same by placing logs after each call, noticed that whenever consumer got stuck, only the logs before createMessageStreams were printed. On Fri, Dec 6, 2013 at 9:40 PM, Jun Rao wrote: > I don't see createMessageStreams in the thread du

Scala 2.10.1 - Issues with starting zookeeper

Hi, I have downloaded kafka-0.8.0-src.tgz, and executed the sbt update, sbt package "++ 2.10.1 package", and the assembly dependency. Modified the scala-version in kafka-run-class.sh to 2.10.1. Upon starting zookeepr,

Re: Embedding a Kafka 0.8 broker for testing

This is something we do routinely. We wrap the KafkaServer from java, it works pretty well. You can use the KafkaServerStartable class, or the KafkaServer class directly (in which case, you might need to do what we did as described in: https://issues.apache.org/jira/browse/KAFKA-1101) On Thu

problem with high-level consumer stream filter regex....

All, I've filed: https://issues.apache.org/jira/browse/KAFKA-1180 We are needing to create a stream selector that essentially combines the logic of the BlackList and WhiteList classes. That is, we want to select a topic that contains a certain prefix, as long as it doesn't also contain a seconda

Re: About speed contorl in Kafka

Sounds to me your consumer performance is the problem, not the producer. So either make your consumers faster or make them consume but drop messages to keep up with the producer speed. This also gives you some means of keeping track of how many messages are lost, why, and when. And just for refere

Re: Topic not created

Thanks Jun for response. So the problem i am facing is expected behavior of kafka. I have just a suggestion that in case of live brokers less than # replicas: There should be topic created so atleast live brokers can receive the data. They can replicate data to other broker once any down broker c

Re: zkClient dependency version with kafka 0.8

It should be 0.3. Where do you see the 0.1 dependency? Guozhang On Thu, Dec 12, 2013 at 1:50 AM, Ted Won wrote: > Hi~, > > In my case 0.3 version is working properly. > On Dec 12, 2013 5:32 PM, "Tarang Dawer" wrote: > > > Hi > > > > Which dependency& version of zkClient , is recommended to be

Re: Topic not created

Currently, when a topic is created, we require # live brokers to be at least # replicas. Once the topic is created, live brokers can be less than # replicas. Thanks, Jun On Thu, Dec 12, 2013 at 6:27 AM, Barto Rael wrote: > Hi All, > > We are having kafka cluster of 2 nodes. (Using Kafka 0.8.0

Re: Embedding a Kafka 0.8 broker for testing

You can take a look at KafkaServerTestHarness in our unit tests. Thanks, Jun On Thu, Dec 12, 2013 at 3:14 AM, Javier Alba wrote: > Hi, > > I would like to embed a Kafka broker in my Java program, for testing > purposes. > > The idea is being capable of running a Junit test that starts a local

Re: About speed contorl in Kafka

I think what you are asking for is backpressure from the broker to the producer. I.e. when the broker got close to full it would start to slow down the producer to let the consumer catch up. This is a fairly typical thing for a message broker to do. Our approach is different though. We have found

Re: why log files are never deleted?

Log deletion is controlled a retention policy, which can be either by time or by size. Currently, there is no purge support yet. Thanks, Jun On Thu, Dec 12, 2013 at 12:55 AM, shahab wrote: > Hi > > I just wonder why the log files, in {kafka_path}/log , are not deleted > automatically? > Is th

Re: About speed contorl in Kafka

You can use more brokers. Another thing is to enable compression in the producer, if you haven't done so. Thanks, Jun On Wed, Dec 11, 2013 at 11:42 PM, xingcan wrote: > Guozhang, > > Thanks for your prompt replay. I got two 300GB SAS disks for each borker. > At peak time, the produce speed fo

Topic not created

Hi All, We are having kafka cluster of 2 nodes. (Using Kafka 0.8.0 version) Replication Factor: 2 Number of partitions: 2 If any one node goes down then topic is not created in kafka. Since we are using two nodes so second node is automatically elected as leader node if down node was leader. Bel

Topic not created

Hi All, We are having kafka cluster of 2 nodes. (Using Kafka 0.8.0 version) Replication Factor: 2 Number of partitions: 2 If any one node goes down then topic is not created in kafka. Since we are using two nodes so second node is automatically elected as leader node if down node was leader. Bel

Re: kafka_2.8.0-0.8.0 maven dependency

Thanks for quick response Joe !! As per my understanding we can use maven dependency of kafka_2.10-0.8.0 in application against binary release of kafka_2.8.0-0.8.0 available on kafka website. On Thu, Dec 12, 2013 at 12:44 PM, Joe Stein wrote: > no plan yet > > you can use the broker binary up

Re: kafka_2.8.0-0.8.0 maven dependency

no plan yet you can use the broker binary up on the download page, same like if you were using the python or c++ client with the broker another scala version is alright for the client On Thu, Dec 12, 2013 at 7:25 AM, Hanish Bansal < hanish.bansal.agar...@gmail.com> wrote: > Hi Joe, > > Is there

Re: kafka_2.8.0-0.8.0 maven dependency

Hi Joe, Is there any planned date to fix this ? If i switch to kafka_2.10-0.8.0 in my application then i will also have to use kafka of same version. For that i'll have to download source of kafka and compile that against scala 2.10, as there is only binary release of kafka_2.8.0-0.8.0 available

Re: kafka_2.8.0-0.8.0 maven dependency

There is an open JIRA for it to fix for now you can switch to org.apache.kafka kafka_2.10 0.8.0 /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop

kafka_2.8.0-0.8.0 maven dependency

Hi All, I am upgrading my system from kafka-0.8.0 beta release to kafka 0.8.0 final release. I am using below maven dependency for kafka 0.8 final release: org.apache.kafka kafka_2.8.0 0.8.0 When i am building my application, kafka 0.8.0 jar is automatically downloaded of size 6 K

Embedding a Kafka 0.8 broker for testing

Hi, I would like to embed a Kafka broker in my Java program, for testing purposes. The idea is being capable of running a Junit test that starts a local kafka broker, and then perform tests on some custom producers and consumers. After searching around, I've found some pieces of code but none of

Re: zkClient dependency version with kafka 0.8

Hi~, In my case 0.3 version is working properly. On Dec 12, 2013 5:32 PM, "Tarang Dawer" wrote: > Hi > > Which dependency& version of zkClient , is recommended to be used with > kafka 0.8 , scala-library 2.8.0 ? > > I see 2 dependencies for zkClient, > 1. com.github.sgroschupf >

Re: Kafka Consumer does not receive any message after a while

Thanks a lot, very good hints. I am trying to see what happened in my case. best, /Shahab On Wed, Dec 11, 2013 at 5:16 PM, Jun Rao wrote: > Have you looked at > > https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Myconsumerseemstohavestopped%2Cwhy%3F > ? > > Thanks, > > Jun > > > On We

why log files are never deleted?

Hi I just wonder why the log files, in {kafka_path}/log , are not deleted automatically? Is there any way to purge those files? Also is there anyway to purge the Kafka queue (make it empty) without having to consuming or knowing the last fetched offset? best, /Shahab

Re: zkClient dependency version with kafka 0.8

Hi Which dependency& version of zkClient , is recommended to be used with kafka 0.8 , scala-library 2.8.0 ? I see 2 dependencies for zkClient, 1. com.github.sgroschupf zkclient 0.1 2 . com.101tec zkclient 0.3 Thanks Tarang Dawer On

Re: zkClient version with Kafka 0.8 & scala-library 2.8.0

Please ignore this email, got submitted accidently.

zkClient dependency version with kafka 0.8

Hi Which dependency & version of zkClient , is recommended to be used with kafka 0.8 , scala-library 2.8.0 ? I see 2 dependencies for zkClient, 1.

zkClient version with Kafka 0.8 & scala-library 2.8.0

Hi which dependency version should i use with kafka 0.8 , 0.1 0.2 or 0.3 ? com.101tec zkclient Also, is this groupID correct or should i use zkClient dependency with groupId