Producer reaches a max of 7Mbps

Hi, I am using kafka producer and broker for a production setup. The expected producer output is 20MBps but I am only getting max of 8MBps. I have verified that we are losing packets by directly connecting to the data source through TCP though the metrics is not reflecting any loss. I went through

Re: ActiveControllerCount is always will be either 0 or 1 in 3 nodes kafka cluster?

There should only be one controller at any point of time in a Kafka cluster. If that controller broker is bounced, then the controller moves to another broker. In general, you want to alert if the addition of the mbean values for ActiveControllerCount across all brokers in a cluster != 1. Thanks,

ActiveControllerCount is always will be either 0 or 1 in 3 nodes kafka cluster?

Hi, there is one Mbean in jconsole with name "kafka.controller":type="KafkaController",name="ActiveControllerCount", the Value of this Mbean is always 1 on one broker and 0 on all others remaining broker of the 3 nodes kafka cluster.Will it always be like this? As per my understanding it will al

Re: kafka.common.OffsetOutOfRangeException

Hi Philip. It looks like this is our case: https://github.com/nathanmarz/storm-contrib/pull/15 It is interesting that the issue is still open ( after more then 1 year) so I am curious how people able to work on production without ability to deploy another topology. Can community please share i

How to link kafka servers

Hi, My endpoint for kafka is in Europe where the a consumers gets the final messages. But I want to write to kafka servers in SG, and the US east and west and will then forward to kafka servers in the UK. How do I do that? Is there a good tutorial? What are the keywords that describe what I wan

Re: Consumer example not working for 0.8 beta

Try producing some new data after the consumer is running. Be default, the consumer only picks up newly produced data. Thanks, Jun On Mon, Nov 18, 2013 at 11:22 AM, Menka wrote: > I am a newbie for Kafka and trying to execute the samples for 0.8 beta > release from > > > https://github.com/ap

Re: kafka.common.OffsetOutOfRangeException

Don't get scared, this if perfectly normal and easily fixed. :-) The second topology attempted to fetch messages from an offset in Kafka that does not exists. This could happen due to Kafka retention policies (messages deleted) or a bug in your code. Your code needs to catch this exception, and the

kafka.common.OffsetOutOfRangeException

We are working with kafka (0.7.2) + storm. 1) We deployed 1st topology which subscribed on Kafka topic and it is working fine already couple of weeks. 2) Yesterday we deploy 2nd topology which subscribed on the same Kafka topic , but 2nd topology immediately failed with exception: *What c

Re: Replica Configuration Settings

Hi Tom, All the replication parameters have some default values: http://kafka.apache.org/documentation.html#brokerconfigs And they are not overridden in the release so default values will be picked, which actually opt to not use replication unless specified per topic since default.replication.fa

Replica Configuration Settings

Hi All, I don't see the replica configuration settings (outlined in the Kafka 0.8 documentation) in the configuration file that comes with the distribution. I was wondering if they are necessary or if they have reasonable defaults? Are there implications for not having them in the configuration fi

Re: High-level consumer not rebalancing.

So I think I got to the root of the problem. Thanks for pointing me in the direction of zookeeper data conflicts. I turned the log level up to INFO and captured a bunch of conflict messages from the zookeeper client. I did an "rmr" on the consumers/ zookeeper node to clear out any lingering data

Re: High-level consumer not rebalancing.

Could you find some entries in the log with the key word "conflict"? If yes could you paste them here? Guozhang On Mon, Nov 18, 2013 at 2:56 PM, Drew Goya wrote: > Also of note, this is all running from within a storm topology, when I kill > a JVM, another is started very quickly. > > Could th

Re: High-level consumer not rebalancing.

Also of note, this is all running from within a storm topology, when I kill a JVM, another is started very quickly. Could this be a problem with a consumer leaving and rejoining within a small window? On Mon, Nov 18, 2013 at 2:52 PM, Drew Goya wrote: > Hey Guozhang, I just forced the error by

Re: ./sbt assembly-package-dependency needed for Kafka 0.7.2?

Joe, Thanks so much. Regards, Menka On Mon, Nov 18, 2013 at 2:13 PM, Joe Stein wrote: > Nope, assembly-package-dependency was introduced for 0.8 > > /*** > Joe Stein > Founder, Principal Consultant > Big Data Open Source Security LLC > http://www.s

Re: High-level consumer not rebalancing.

Hey Guozhang, I just forced the error by killing one of my consumer JVMs and I am getting a consumer rebalance failure: 2013-11-18 22:46:54 k.c.ZookeeperConsumerConnector [ERROR] [bridgeTopology_host-1384493092466-7099d843], error during syncedRebalance kafka.common.ConsumerRebalanceFailedExceptio

Re: High-level consumer not rebalancing.

Hello Drew, Do you see any rebalance failure exceptions in the consumer log? Guozhang On Mon, Nov 18, 2013 at 2:14 PM, Drew Goya wrote: > So I've run into a problem where occasionally, some partitions within a > topic end up in a "none" owner state for a long time. > > I'm using the high-leve

High-level consumer not rebalancing.

So I've run into a problem where occasionally, some partitions within a topic end up in a "none" owner state for a long time. I'm using the high-level consumer on several machines, each consumer has 4 threads. Normally when I run the ConsumerOffsetChecker, all partitions have owners and similar l

Re: ./sbt assembly-package-dependency needed for Kafka 0.7.2?

Nope, assembly-package-dependency was introduced for 0.8 /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop

./sbt assembly-package-dependency needed for Kafka 0.7.2?

Do we need to run ./sbt assembly-package-dependency for Kafka 0.7.2? I am running into this. [idhuser@vmidh kafka-0.7.2-incubating-src]$ ./sbt assembly-package-dependency [info] Building project Kafka 0.7.2 against Scala 2.8.0 [info]using KafkaProject with sbt 0.7.5 and Scala 2.7.7 *[error]

Re: Mirror Maker [Aggregate cluster]

You can also just use a different namespace on the same zookeeper cluster if both local and aggregate are in the same colo. Thanks, Neha On Mon, Nov 18, 2013 at 12:44 PM, Vinicius Carvalho < viniciusccarva...@gmail.com> wrote: > Hi there, I'm sorry about this silly question, just trying to make

Mirror Maker [Aggregate cluster]

Hi there, I'm sorry about this silly question, just trying to make sure I got this right: Having a separate cluster for aggregate, means that I would need a second zookeeper for that cluster right? I would need a ZK for my local cluster, and then a ZK for my aggregate, is this correct? Regards

Re: Kafka cluster with lots of topics

If I understand your comment correctly you are trying to use kafka topics as per-endpoint message queues. I may be mistaken, but to me Kafka seems does not really seem like a good match for that. For such a purpose you will eventually want something that is not actually a queue - a means to

Consumer example not working for 0.8 beta

I am a newbie for Kafka and trying to execute the samples for 0.8 beta release from https://github.com/apache/kafka/tree/0.8.0-beta1candidate1/examples/src/main/java/kafka/examples I tried KafkaCons