Rebalance Logs

Where in the broker logs can I see that a rebalance is happening? Will the state change log tell me this?

Re: Question about manual tracking of Offset

Thanks, Jun. Greatly helps and that was simple solution which I overlooked :) Regards, KR On Tue, Mar 25, 2014 at 9:03 AM, Jun Rao wrote: > While you are consuming messages, you should use > MessageAndOffset.nextOffset() when saving offsets. > > Thanks, > > Jun > > > On Mon, Mar 24, 2014 at 1

RE: Separate broker replication traffic from producer/consumer traffic

Hi Jay, We're pushing a lot of data from the producers (n) and have many consumers (3n) reading them. We're configured to have replication factor of 3, so replication traffic is about (2n). Currently all traffic was on a single NIC, so that's about (6n) total. Having the replication traffic o

Re: Separate broker replication traffic from producer/consumer traffic

Hey Otto, Yeah this isn't something we've really thought about. Presumably the implementation would be that the server accept connections on two interfaces. That is pretty easy. However the harder part is that I think this would require updating the metadata to advertise a different ip/host to oth

Re: producers limit

Hi Neha, Thank you for the numbers as this information helped me to size my project as well. While we are on this topic, do you mind commenting on the maximum number of topics that can be hosted on a 8-10 node cluster. We have a need for about 100k topics and similar number of producers and

Re: Separate broker replication traffic from producer/consumer traffic

Hi Otto, We've separated our traffic for a couple of reasons: 1. We wanted to protect our producer bandwidth to maintain a low latency pipeline 2. We expected consumers to sometimes pick up from an older offset and clog the pipe, causing latency for other services 3. When an out of sync replica co

Kafka contrib/hadoop-consumer not working with hadoop 2.2.0

Kafka native hadoop-consumer is compiled with hadoop 0.20 dependencies, so isn't working with hadoop last releases (hadoop 2.2.0) The code hasn't support from two years ago. Is this a discarded option to load kafka messages to HDFS? Do you recommend other solution like Camus? Thanks in advance.

Re: Kafka contrib/hadoop-consumer not working with hadoop 2.2.0

I wrote my own hdfs writer using threaded simple kafka consumer writing to hdfs using java FSDataOutputStream works perfect mike marzo 908 209-4484 On Mar 26, 2014 12:59 PM, "Marcelo Valle" wrote: > Kafka native hadoop-consumer is compiled with hadoop 0.20 dependencies, so > isn't working with

Re: When is the status of isr updated

Hello Jie, Yes, the ISR shrinking is handled in a background thread which will periodically check the status of each replica and probably kick them out of ISR if they are not in-sync with the leader. You can tune the frequency in this broker config: replica.lag.time.max.ms The ISR expanding is h

Re: Rebalance Logs

Hi Tom, You can find the rebalance logs in the controller log and the stata change log. Generally you should be able to see the decision being made about change leaders on the controller log, and receiving controller request to change to leader/follower in other brokers' state change log. Guozhan

data loss on replicated topic

My company currently testing Kafka for throughput and fault tolerance. We've set up a cluster of 5 Kafka brokers and are publishing to a topic with replication factor 3 and 100 partitions. We are publishing with request.required.acks == -1 (e.g. All ISR replicas must ACK before the message is c

Re: data loss on replicated topic

I just saw https://issues.apache.org/jira/browse/KAFKA-1193 which seems like it could be the cause of this. Does that sound right? Is there a patch we can test? Any date/time when this is expected to be fixed? From: New User mailto:od...@3cinteractive.com>> Date: Wednesday, March 26, 2014 at 11:

RE: Separate broker replication traffic from producer/consumer traffic

Jay, Thanks for your responses. What type of hardware specs are your Kafka servers? 10G or bonded NICs? Joris, Thank you for your detailed example. I was thinking along those lines before, and was hoping for something cleaner. Glad to know that it actually works! That's a whole lot of host f

Help in setting up Kafka cluster

Hi, First time I am trying to setup new kafka cluster. I have tried sudo cluster with cli based kafka producer and consumer. Having difficulties with setting up log aggregation kafka producer and consumer. I would appreciate if anyone can help me in this. - roy

Re: Help in setting up Kafka cluster

Hi Roy, I wonder if you were able to start the broker following the steps here: http://kafka.apache.org/documentation.html#quickstart That page also shows you how to create a topic and send/consume messages using the console producer/consumer. Let us know if you run into any problems, Tim On

Re: When is the status of isr updated

Hi Jie, If you see frequent changes in the ISR for some partitions, you can follow the recommendation hereto address that. Thanks, Neha On Wed, Mar 26, 2014 at 11:11 AM, Guozhang Wang

How are rebalance failures raised to consumers?

The pattern for creating and operating consumers that we use is to create the consumer connector, create the streams and then consume each stream by waiting on the iterator. If a rebalance occurs and fails, how is the error raised to the consumer? Will I get an exception while waiting on the itera

Re: producers limit

I have a possibly related question: In a batch sending mode it's possible that messages are lost if producer dies while accumulating the batch. What is the performance overhead if we don't use batching? Thanks. On Wed, Mar 26, 2014 at 9:24 AM, Jagbir wrote: > Hi Neha, > > Thank you for the nu

Re: How are rebalance failures raised to consumers?

Currently, there is no callback on rebalance failure. The consumer will retry failed rebalances. If all retries fail, we just log the error. Thanks, Jun On Wed, Mar 26, 2014 at 5:01 PM, Tom Amon wrote: > The pattern for creating and operating consumers that we use is to create > the consumer