Re: Consumer Parallelism

Hi Guozhang, Thank you! Could I say the consumer 'take turns to consume' is resulted by the correspond partition got the 'message write'? The problem I am facing is my 'enrichment' (getting more data based on raw data) consumer took too much time to complete one message consumption. To explor

Re: Consumer Parallelism

Hi Guozhang, I think what I am looking for is the real 'randomness' when producer write to the partitions. Based on my log, through a long time period, only one partition got the write, while the other side, only one consumer is active. In my case the consumer is slow, so when it comes back for th

Re: Issue with 240 topics per day

Todd -- can you share details of the ZK cluster you are running, to support this scale? Is it one single Kafka cluster? Are you using 1 single ZK cluster? Thanks, Philip   - http://www.philipotoole.com On Monday, August 11, 2014 9:32 PM, Todd Palino

kafka TestUtils createBrokerConfig issue

Trying to write a unit test case for Kafka, and stuck with strange createBrokerConfig issue. on TestUtils.createBrokerConfigs(1) it gives me compilation error The method createBrokerConfigs(int, boolean) in the type TestUtils is not applicable for the arguments (int) When I looked into the scala

RE: Kafka Consumer not consuming in webMethods.

We will work on upgrade. Thanks, Balaji From: Jun Rao [mailto:jun...@gmail.com] Sent: Monday, August 11, 2014 10:30 PM To: Seshadri, Balaji Cc: users@kafka.apache.org Subject: Re: Kafka Consumer not consuming in webMethods. 0.8-beta is really old. Could you try using 0.8.1.1? Thanks, Jun On

Re: Consumer Parallelism

I see your question now. You may want to read this FAQ: https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Whyisdatanotevenlydistributedamongpartitionswhenapartitioningkeyisnotspecified ? On Tue, Aug 12, 2014 at 8:11 AM, Mingtao Zhang wrote: > Hi Guozhang, > > I think what I am looking

Re: LeaderNotAvailableException

The "0" there in the kafka-topics output is the broker id. >From the broker log I think you are hitting KAFKA-1393 , which Kafka version are you using? Guozhang On Mon, Aug 11, 2014 at 10:37 PM, Ryan Williams wrote: > Thanks for the heads up o

Re: kafka TestUtils createBrokerConfig issue

Are you using it from java? The generated class has only the following signature on createBrokerConfig. javap core/build/classes/test/kafka/utils/TestUtils |grep -i createbrokercon Warning: Binary file core/build/classes/test/kafka/utils/TestUtils contains kafka.utils.TestUtils public static jav

Re: kafka TestUtils createBrokerConfig issue

Parin, I only see the following two functions in TestUtils: createBrokerConfigs(Int, Boolean = true) createBrokerConfig(Int, Int = choosePort(), Boolean = true) // this one does not have the same function name Guozhang On Tue, Aug 12, 2014 at 8:14 AM, Parin Jogani wrote: > Trying to write a

Re: Consumer Parallelism

Great! I am in the 10 min category for sure. I do see there is NO partition key provided in our code. I feel it's too much 'customization' when Kafka provides a 'randomness' default partition strategy while have another layer doing the 10 min tricky to optimize socket stuff. Anyway, thank you

Re: Consumer Parallelism

Mingtao, We have also noticed this issue and are trying to fix it in the new producer: KAFKA-1586 Guozhang On Tue, Aug 12, 2014 at 9:41 AM, Mingtao Zhang wrote: > Great! I am in the 10 min category for sure. I do see there is NO > partition

Re: LeaderNotAvailableException

Using version 0.8.1. Looking to update to 0.8.1.1 now probably. On Tue, Aug 12, 2014 at 9:25 AM, Guozhang Wang wrote: > The "0" there in the kafka-topics output is the broker id. > > From the broker log I think you are hitting KAFKA-1393 > , wh

Using the kafka dissector in wireshark/tshark 1.12

I'd seen references to there being a Kafka protocol dissector built into wireshark/tshark 1.12, but what I could find on that was a bit light on the specifics as to how to get it to do anything -- at least for someone (like me) who might use tcpdump a lot but who doesn't use tshark a lot.

Blocking Recursive parsing from kafka.consumer.TopicCount$.constructTopicCount

Hi All, We have a typical cluster of 3 kafka instances backed by 3 zookeeper instances (kafka version 0.8.1.1, scala version 2.10.3, java version 1.7.0_65). On consumer end, when some of our consumers were getting recycled, we found a troubling recursion which was taking a busy lock and blocking

Correct way to handle ConsumerTimeoutException

Folks, I am using consumer.timeout.ms to force a consumer jump out hasNext call, which will throw ConsumerTimeoutException. It seems that upon receiving this exception, the consumer is no longer usable and I need to call .shutdown, and recreate: try{ } catch (ConsumerTimeoutException ex) { logge

Re: Blocking Recursive parsing from kafka.consumer.TopicCount$.constructTopicCount

Hi Jagbir, The thread dump you uploaded is not readable, could you re-parse it and upload again? Guozhang On Tue, Aug 12, 2014 at 11:48 AM, Jagbir Hooda wrote: > Hi All, > We have a typical cluster of 3 kafka instances backed by 3 zookeeper > instances (kafka version 0.8.1.1, scala version 2.

Re: Correct way to handle ConsumerTimeoutException

Hi Chen, The rational of using the consumer timeout exception is to indicate when there is no more data to be consumed, and hence upon capturing the exception the consumer should be closed. If you want to restart the consumer in handling the timeout exception, then you should probably just increa

Strange topic-corruption issue?

[ "Aha!", you say, "now I know why this guy's been doing so much tshark stuff!" (-: ] Hi. I'm running into a strange situation, in which more or less all of the topics on our Kafka server behave exactly as expected... but the data produced by one family of applications is producing fairly

RE: Blocking Recursive parsing from kafka.consumer.TopicCount$.constructTopicCount

> Date: Tue, 12 Aug 2014 16:35:35 -0700 > Subject: Re: Blocking Recursive parsing from > kafka.consumer.TopicCount$.constructTopicCount > From: wangg...@gmail.com > To: users@kafka.apache.org > > Hi Jagbir, > > The thread dump you uploaded is not readable, could you re-parse it and > upload agai