Re: Describe command does not update details about killed brokers

Yes that problem is similar to mine. My problem is to list offline partitions. and when i describe an offline topic it shows leader and isr value to be the last live broker (even after it is killed. (ie) now there is no live brokers at all.) i think this is because zk returns value based on last li

Re: Possible DEAD LOCK for one day at broker controller?

Hi Kishore Senji, Did you constantly send messages to your test topic? Or just one time send? I've just did some test, the log.lastModified is updated with every message received (or every flush to disk at least). So I think if your interval between two neibouring messages is never smaller than re

KafkaConsumer from trunk hangs indefinitely on poll

Hi, I'm trying to test the new java KafkConsumer (for use with 0.8.3) and have pulled the code from trunk, but for me the following just ends up hanging indefinitely. Map props = new HashMap(); props.put("bootstrap.servers", "broker1:2181"); props.put("key.deserializer», "org.apache.kafka.co

Zookeeper use cases with Kafka

Hello Folks, I wish to contribute to Kafka internals. And, one of the things which can help me do that is understanding how kafka uses zookeeper. I have some of these basic doubts:- 1. Is zookeeper primarily used for locking ? If yes, in what cases and what kind of nodes does it use - sequential/

Zookeeper jmx monitoring for kafka

Hi, Zookeeper monitoring - I was going through the 'Monitoring' section in http://kafka.apache.org/coding-guide.html I have setup Ganglia to monitor all the stats inocming from kafka jmx port But, in addition, there are a bunch of stats that Zookeeper also exposes in its jmx port. They are docume

Re: Zookeeper use cases with Kafka

Hi Prabcs, Much of that information can be found in the documentation and on the wiki. The remaining can be found in the code. Any improvements to the documentation is not only welcome but encouraged. Below are a few links to get you started: Documentation (See "Zookeeper Directories"): http://ka

Re: Zookeeper use cases with Kafka

Hi, 1. ZK is used for co-ordination between brokers, controller election, leader election, storing topic configuration etc. I think we use both sequential and ephemeral nodes. 2. Yes, Kafka uses ZK watches for controller changes, new topic creation, new partition creation, leader chan

Re: Reduce latency

Thank you Jay, that really helps! Kishore, Where you can monitor whether the network is busy on IO in visual vm? Thanks. I am running 90 producer process on 90 physical machines in the experiment. On Tue, Aug 18, 2015 at 1:19 AM, Jay Kreps wrote: > Yuheng, > > From the command you gave it looks

KafkaConsumer from 0.8.3 trunk hangs indefinitely on poll

Hi, I'm trying to test the new java KafkConsumer in 0.8.3 and have pulled the code from trunk, but for me the following just ends up hanging indefinitely. Map props = new HashMap(); props.put("bootstrap.servers", "broker1:2181"); props.put("key.deserializer», "org.apache.kafka.common.serializ

Is Kafka suitable for high volume, real-time sessions ?

Hi All, I wonder if I could get some feedback on a proposed design involving Kafka for high volume, real-time sessions ? See the diagram below for the proposed scheme. Some additional information: There are a number of different messages streams (>10), with different throughput and lat

Re: KafkaConsumer from 0.8.3 trunk hangs indefinitely on poll

As you can see in the javadoc for KafkaConsumer, you need to call poll() in a loop. Something like: while (true) { * ConsumerRecords records = consumer.poll(100); * records.forEach(c -> queue.add(c.value())); * * } On Tue, Aug 18, 2015 at 2:46 AM, Krogh-Moe, Espen wrote: >

Re: Possible DEAD LOCK for one day at broker controller?

Yes you are right. I misread the code. So the only thing that can explain the behavior you are seeing is that may be there are many segments that need to be deleted all at once. Can you try may be reducing the retention.ms in smaller intervals - like reduce it to 9 days from 10 days and see if the

Re: Error handling in New AsyncProducer

kishore How is going to reduce throughput if we have enough memory for batch size ?, could you please explain On Tue, Aug 18, 2015 at 11:47 AM, Kishore Senji wrote: > But this will reduce the throughput in a good scenario. May be we need to > enhance the Callback interface appropriately. > > On

Patch for SSL setup for Kafka

Hi, Has anyone used patch provided in below JIRA to setup SSL for Kafka? https://issues.apache.org/jira/browse/KAFKA-1690 I see there are many pathces. which one is stable? If anyone has instructions on how o use it, it will help. LCassa.

Re: [DISCUSSION] Kafka 0.8.2.2 release?

Thanks Jun. I updated the list with your suggestions. If anyone feels we are missing a critical patch for 0.8.2.2, please speak up. Gwen On Mon, Aug 17, 2015 at 5:40 PM, Jun Rao wrote: > Hi, Grant, > > I took a look at that list. None of those is really critical as you said. > So, I'd suggest

Re: Error handling in New AsyncProducer

If you have enough memory for batch size then the scenario is no different than your original question because block on full memory is true by default (and if it is false you actually get a notification via exception). So when you said reduce the buffer memory to "minimum", I assumed you are referr

Re: Patch for SSL setup for Kafka

Hi ,     you can use the latest diff from the review board here  https://reviews.apache.org/r/33620/diff/17/ . Attached patches on the JIRA have some issues in cleanly applying against the trunk. You can use git apply. For Instructions I’ve added this wiki  https://cwiki.apache.org/confluence/disp

Re: KafkaConsumer from trunk hangs indefinitely on poll

It looks like you might have bootstrap servers pointed to zookeeper. It should point to the brokers instead since the new consumer doesn't use zookeeper. As for the hanging, it is a known bug that we're still working on. -Jason On Tue, Aug 18, 2015 at 3:02 AM, Krogh-Moe, Espen wrote: > Hi, > >

Re: [DISCUSSION] Kafka 0.8.2.2 release?

How about looking at the scope for the 0.8.3 release first before we cut yet another point release off of 0.8.2.2? Each release includes some overhead and if there is a larger release in the works, it might be worth working on getting that. My take is that the 2 things the community has been waitin

Re: [DISCUSSION] Kafka 0.8.2.2 release?

I completely agree that community focus on 0.8.3 is important and don't want to take away from that effort. That said, not everyone will upgrade to 0.8.3 immediately (unfortunately), and there is still value in releasing bugfixes on previous releases. I'm keeping the scope very small, to make the

Re: [DISCUSSION] Kafka 0.8.2.2 release?

I am strongly in favor of cutting a 0.8.2.2 release, but I don’t think that it needs to include anything other than the fix for Snappy that kicked off this discussion in the first place. If there are additional critical issues that can be included without delaying the process, I see no downside.

Re: [DISCUSSION] Kafka 0.8.2.2 release?

I sort of follow Daniel Nelson on this issue: cut a 0.8.2.2 but not include much thing besides Snappy fixes. I mean, include a couple of additional critical bug fixes, if really urgent, and that's it. On Tue, Aug 18, 2015 at 3:25 PM, Daniel Nelson wrote: > I am strongly in favor of cutting a 0.8

Re: Reduce latency

Also, When I set the target throughput to be 1 records/s, The actual test results show I got an average of 579.86 records per second among all my producers. How did that happen? Why this number is not 1 then? Thanks. On Tue, Aug 18, 2015 at 10:03 AM, Yuheng Du wrote: > Thank you Jay, tha

Re: Reduce latency

Hi Yuheng, The 1 record/s is just a param for producerperformance for your producer target tput. It only takes effect to do the throttling if you tries to send more than 1 record/s. The actual tput of the test depends on your producer config and your setup. -Tao On Tue, Aug 18, 2015 at

Re: Reduce latency

I see. Thank you Tao. But now I don't get it what Jay said that my latency test only makes sense if I set a fixed throughput. Why do I need to set a fixed throughput for my test instead of just set the expected throughput to be -1 (as much as possible)? Thanks. On Tue, Aug 18, 2015 at 2:43 PM, Ta

Re: [DISCUSSION] Kafka 0.8.2.2 release?

I agree: keep it simple :) The latest stable version of Kafka right now has a critical bug in it. Fixing that would be good enough. 0.8.2.2 should probably just a maintenance/bugfix release. > On Aug 18, 2015, at 14:29, Edward Ribeiro wrote: > > I sort of follow Daniel Nelson on this issue

Re: [DISCUSSION] Kafka 0.8.2.2 release?

Jun, KAFKA-2147 doesn't seem to have a commit associated with it, so I can't cherrypick just this fix. I suggest leaving this out since there is a 0.8.2.x workaround in the JIRA. Gwen On Mon, Aug 17, 2015 at 5:24 PM, Jun Rao wrote: > Gwen, > > Thanks for putting the list together. > > I'd reco

Re: Reduce latency

>From what I understand, if you set the throughput to -1, the producerperformance will push records as much as possible to an internal per topic per partition queue. In the background there is a sender IO thread handling the actual record sending process. If you push record to the queue faster than

Re: Reduce latency

I see. So the internal queue overwrites the producer buffer size configuration? When buffer is full the producer will block sending, right? On Tue, Aug 18, 2015 at 3:52 PM, Tao Feng wrote: > From what I understand, if you set the throughput to -1, the > producerperformance will push records as m

Re: [DISCUSSION] Kafka 0.8.2.2 release?

Any objections if I leave KAFKA-2114 (setting min.insync.replicas default) out? The test code is using changes that were done after 0.8.2.x cut-off, which makes it difficult to cherry-pick. Gwen On Tue, Aug 18, 2015 at 12:16 PM, Gwen Shapira wrote: > Jun, > > KAFKA-2147 doesn't seem to have

Re: Reduce latency

you can configure that, to block or to fail: http://kafka.apache.org/documentation.html#producerconfigs By default it should block On Tue, Aug 18, 2015 at 4:57 PM, Yuheng Du wrote: > I see. So the internal queue overwrites the producer buffer size > configuration? When buffer is full the produc

spark broadcast variable of Kafka producer throws ConcurrentModificationException

Hi, Did anyone see java.util.ConcurrentModificationException when using broadcast variables? I encountered this exception when wrapping a Kafka producer like this in the spark streaming driver. Here is what I did. KafkaProducer producer = new KafkaProducer(properties); final Broadcast bCastProduce

Re: [DISCUSSION] Kafka 0.8.2.2 release?

Yes, I think it's fine to leave out both KAFKA-2147 and KAFKA-2114. Thanks, Jun On Tue, Aug 18, 2015 at 12:59 PM, Gwen Shapira wrote: > Any objections if I leave KAFKA-2114 (setting min.insync.replicas default) > out? > > The test code is using changes that were done after 0.8.2.x cut-off, whi

Re: Patch for SSL setup for Kafka

Thank You! I will look into it. On Tue, Aug 18, 2015 at 10:34 AM, Sriharsha Chintalapani wrote: > Hi , > you can use the latest diff from the review board here > https://reviews.apache.org/r/33620/diff/17/ . Attached patches on the > JIRA have some issues in cleanly applying against the trun

How to make KafkaStream iterator blocking?

Hello, I saw this thread and have the opposite problem. I am using the code here

Re: spark broadcast variable of Kafka producer throws ConcurrentModificationException

Hi, looks like the exception is occurring at kryo serialization. make sure you are not concurrently modifying java.util.Vector data structure. kumar On Wed, Aug 19, 2015 at 3:32 AM, Shenghua(Daniel) Wan wrote: > Hi, > Did anyone see java.util.ConcurrentModificationException when using > broa

Using multiple consumers for same topic

Hello, If I use multiple consumers , each running on a different machine, for a single topic, will the output get duplicated ? Regards, Saurav

Re: Using multiple consumers for same topic

Basically, if a source record is already consumed by Consumer1, will it also get consumed by Consumer2 and Consumer3 all subscribing to the same topic ? Regards, Saurav Datta Data Engineer| Desk - (408)967-7360| Cell - (408)666-1722 From: , Saurav Datta mailto:sda...@paypal.com>> Date: Tuesda