Re: benchmark kafka on 10GbE network

I have 8 partitions / 1 topic. Do you mean hack the new producer to add more threads ? The current implementation manages a single send thread. Thanks, Manu On Wed, Nov 26, 2014 at 1:28 AM, Jay Kreps wrote: > Yeah, neither of those are simple to optimize. The CRC is already the > optimized java

Re: isr never update

my kafka version is kafka_2.10-0.8.1.1.jar *state-change log:* [2014-11-25 02:30:19,290] TRACE Controller 29 epoch 7 sending UpdateMetadata request (Leader:29,ISR:29,24,LeaderEpoch:10,ControllerEpoch:4) with correlationId 1803 to broker 20 for partition [org.nginx,32] (state.change.logger) *cont

Kafka 0.8.2 log cleaner

I’m wondering if you could tell me whether log cleaner implemented in 0.8.2 because it seems like it didn’t. Thanks John

Does adding ConsumerTimeoutException make the code more robust?

Hi, I am just wondering if the below snippet ConsumerIteratori.e. by setting a consumer timeout, catching it and again just waiting for the next message make it more robust? Regards, Rahul.

how many brokers to set in kafka

Hi, all We are having 3 production server to setup for kafka cluster, I wonder how many brokers to configure for each server. thanks -- Alec Li

Re: how many brokers to set in kafka

Is there any rules to determine or optimize the number of brokers? On Thu, Nov 27, 2014 at 11:09 AM, Sa Li wrote: > Hi, all > > We are having 3 production server to setup for kafka cluster, I wonder how > many brokers to configure for each server. > > > thanks > > > -- > > Alec Li > -- Al

Re: kafka web console running error

I am using https://github.com/claudemamo/kafka-web-console version, and do you mind to tell where about to make such modification? thanks Alec On Mon, Nov 24, 2014 at 11:16 PM, Yang Fang wrote: > do you see error msg "Too many open files"? it tips you should modify > nofile > > On Tue, Nov 25,

maximum number of open file handles

Hi, all I read the comments from http://www.michael-noll.com/tutorials/running-multi-node-storm-cluster/, Michael mentioned to increase the maximum number of open file handles for the user kafka to 98,304 (change kafka to whatever user you are running the Kafka daemons with – this can be your own

Re: Partition id of consumed message?

It's a field of MessageAndMetadata actually but there won't be difference from Scala's view. On Thu, Nov 27, 2014 at 9:57 AM, Palur Sandeep wrote: > Thank you. but I dont see that method. I dont know whats wrong > > On Wed, Nov 26, 2014 at 6:31 PM, Jun Rao wrote: > > > Yes, the consumer iterato

Re: maximum number of open file handles

This is not an absolute rule/number. It depends on your Kafka usage and your existing limits. SPM will show you the number of open files Kafka is using, as will a tool like 'lsof'. Otis -- Monitoring * Alerting * Anomaly Detection * Centralized Log Management Solr & El