Re: Kafka Monitoring

Jun. Greatly appreciated. On Wed, Sep 4, 2013 at 10:12 PM, Jun Rao wrote: > Updated the doc at http://kafka.apache.org/documentation.html#monitoring > > Hopefully that answers your questions. > > Thanks, > > Jun > > > On Tue, Sep 3, 2013 at 11:16 PM, Vadim Keylis > wrote: > > > Good evening. I

Re: Kafka Monitoring

Updated the doc at http://kafka.apache.org/documentation.html#monitoring Hopefully that answers your questions. Thanks, Jun On Tue, Sep 3, 2013 at 11:16 PM, Vadim Keylis wrote: > Good evening. I have read through section of monitoring. I tried to map > each section to corresponding JMX attri

Re: Sometimes get no message streams in consumer

What createMessageStreams() returns should match what's in the input. Is it possible for getTopicMap() to return an empty map? Thanks, Jun On Wed, Sep 4, 2013 at 1:43 PM, Ian Friedman wrote: > Hey gang, I have a strange intermittent issue with our 0.7.2 consumers > that is perpelxing me: > >

Re: Kafka-0.8.0-beta1-src Has ObjectName starting with Double Quotes

Those quotes are added by the metrics package that we use. Thanks, Jun On Wed, Sep 4, 2013 at 7:34 AM, Rajasekar Elango wrote: > We have the same problem, it doesn't work with restful JMX console > jiminix. > Is it possible to change kafka to expose mbeans wi

Sometimes get no message streams in consumer

Hey gang, I have a strange intermittent issue with our 0.7.2 consumers that is perpelxing me: It seems like, sometimes very infrequently, when restarting our Consumer fleet, some consumers threads just fail to start. The consumer connector itself starts fine, and seems to be involved in subsequ

Re: problems reasignating partitions

I have few questions before I can help you with your issue - 1. Which version/build of Kafka are you using? 2. Is the current status that you provided below before or after issuing the reassign replica command? 3. If the answer to 2 above is "after issuing the command", how long was it after which

Re: default producer to retro-fit existing log files collection process?

commons-logging has a log4j logger, so perhaps you just need to use it and the log4j-kafka appender to achieve your goal? On Tue, Sep 3, 2013 at 2:08 PM, Maxime Petazzoni wrote: > Tomcat uses commons-logging for logging. You might be able to write an > adapter towards Kafka, in a similar way as

Re: default producer to retro-fit existing log files collection process?

As Neha says the best thing we currently provide is console producer. Providing a more flexible framework specifically targeted at log slurping would be a cool open source project. -Jay On Wed, Sep 4, 2013 at 7:34 AM, Neha Narkhede wrote: > Quick and dirty solution would be to somehow tail the

Re: Number of file handles increases indefinitely in producer if broker host is unresolvable

Sure. Thanks Neha. Created Issue: https://issues.apache.org/jira/browse/KAFKA-1041 On Wed, Sep 4, 2013 at 10:32 AM, Neha Narkhede wrote: > Ideally if the producer runs into any error, it should close the previous > socket and open a new one. Seems like that is not happening here. I will > tak

RE: default producer to retro-fit existing log files collection process?

Quick and dirty solution would be to somehow tail the logs and use console producer to send the data to kafka. Thanks, Neha On Sep 3, 2013 2:09 PM, "Maxime Petazzoni" wrote: > Tomcat uses commons-logging for logging. You might be able to write an > adapter towards Kafka, in a similar way as the

Re: Kafka-0.8.0-beta1-src Has ObjectName starting with Double Quotes

We have the same problem, it doesn't work with restful JMX console jiminix. Is it possible to change kafka to expose mbeans without quotes? Thanks, Raja. On Wed, Sep 4, 2013 at 10:30 AM, Neha Narkhede wrote: > I had the same problem and could not use jmxterm t

Re: Number of file handles increases indefinitely in producer if broker host is unresolvable

Ideally if the producer runs into any error, it should close the previous socket and open a new one. Seems like that is not happening here. I will take a closer look at this today. Do you mind filing a bug? Thanks, Neha On Sep 4, 2013 7:23 AM, "Rajasekar Elango" wrote: > I can easily reproduce t

Re: Kafka-0.8.0-beta1-src Has ObjectName starting with Double Quotes

I had the same problem and could not use jmxterm to inspect jmx beans due to this issue. But we tried escaping the quotes and it works with our internal monitoring system as well as JmxTool that ships with Kafka. Thanks, Neha On Sep 4, 2013 7:16 AM, "Monika Garg" wrote: > Hi, > > Kafka-0.8.0-bet

Re: Sending huge binary files via Kafka?

The message size limit is imposed to protect the brokers and consumers from running out of memory. The consumer does not have support for streaming a message and has to allocate memory to be able to read the largest message. You could try compressing the files but I'm not sure if that will get you

Kafka-0.8.0-beta1-src Has ObjectName starting with Double Quotes

Hi, Kafka-0.8.0-beta1-src is having doublequotes in the start of Objects name obtained from jConsole.Due to this I am not able to use jmxTrans to monitor my kafka-0.8.0 cluster. Please help in solving the issue. Regards Monika Garg Associate Software Engineer Impetus Infotech (India) Pvt. Lt

Re: Number of file handles increases indefinitely in producer if broker host is unresolvable

I can easily reproduce this with console producer, If I run console producer with right hostname and if broker is not running, the console producer will exit after three tries. But If I run console producer with unresolvable broker, it throws below exception and continues to wait for user input, e

Re: zookeeper version issue

Kafka 0.8.0 builds with "org.apache.zookeeper" % "zookeeper" % "3.3.4" /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop ***

Re: zookeeper version issue

I found the same problem. Kafka 08 is bundled with zookeeper 3.3.3, which is causing some issues

Sending huge binary files via Kafka?

Hello, I have a proposal for an architecture on my desk, where people want to store huge binary files (like images and videos up to a size of several GB) in RiakCS. But the connection to RiakCS is supposed to work through Apache Kafka, so there will be a Kafka producer fetching the files from the