Re: Changing the Producer Config at runtime and TCP connections

Each producer has its own TCP connections. The socket related configs for the producer are listed in http://kafka.apache.org/documentation.html#producerconfigs Currently, we don't support changing the configs in the producer dynamically. You will have to create new producers to pick up config chan

Re: trying to run Kafka_2.9.2-0.8.1.1 on windows system, getting errors

We fixed the windows scripts in trunk. However, we didn't backport it to 0.8.1.1. Could you try the windows scripts in trunk? Thanks, Jun On Wed, Apr 30, 2014 at 2:51 PM, David Novogrodsky < david.novogrod...@gmail.com> wrote: > I am trying to run Kafka on the company windows system. I untare

Re: Kafka metrics: Bytesout greater than Bytesin??

In 0.8, we have a MaxLag jmx metrics to monitor the consumer lag. In 0.7, we only have a command line tool ConsumerOffsetChecker. Thanks, Jun On Wed, Apr 30, 2014 at 7:17 AM, Arnaud Lawson wrote: > Jun, > > Thanks for the pointer to the docs. It would be great if more detail like > this discus

Re: Kafka/Zookeeper co-location

Hi, We have Kafka + ZK co-located on EC2 instances. Check this thread from a few weeks ago: http://search-hadoop.com/m/4TaT4Sb6Jg2 Otis -- Performance Monitoring * Log Analytics * Search Analytics Solr & Elasticsearch Support * http://sematext.com/ On Thu, Apr 24, 2014 at 11:53 AM, Andrew

Changing the Producer Config at runtime and TCP connections

Hi Kafka User and Dev team, I do not see any API to change the producer configuration dynamically at run-time after the producer object has been created. We have situation where our pool is deployed in private cloud (open stack VMs), and we need to manage the lifecycle of the Producer respect

Re: kafka 0.8.1.1 for scala 2.8.2 is missing

Yes this is "intentional" in that the scaladoc gradle task for 2.8.2 ran into errors and we couldn't figure it out. The initial proposal was to drop support for 2.8.x altogether since most people have moved on to 2.9/2.10 but we decided to keep 2.8.0 since that was building fine. I thought 2.8.0 is

kafka 0.8.1.1 for scala 2.8.2 is missing

Hi Is this intentional? Otherwise, could you release this? Thank you Best, Jae

Re: Kafka Mock

Thanks Clark. -Matt On Apr 30, 2014, at 11:04 AM, Clark Breyman wrote: > Matt - > Here's a small Gist that shows some of the setup for > Consumer.createJavaConsumer(). > https://gist.github.com/clarkbreyman/548f25ad78c4bb4a7df7 > > -Clark > > > On Wed, Apr 30, 2014 at 7:33 AM, Matt Narrell wr

trying to run Kafka_2.9.2-0.8.1.1 on windows system, getting errors

I am trying to run Kafka on the company windows system. I untared the binaries on the C drive. When I try to run zookeeper I get this error. C:\kafka_2.9.2-0.8.1.1>cd bin C:\kafka_2.9.2-0.8.1.1\bin>cd windows C:\kafka_2.9.2-0.8.1.1\bin\windows>zookeeper-server-start config\zookeeper.prope rtie

Re: Need example of robust Producer

Does this help: https://github.com/stealthly/scala-kafka/blob/master/src/main/scala/KafkaPr oducer.scala -Dragos On 4/30/14, 5:29 AM, "Chris Helck" wrote: >I am new to Kafka. I'd like to see example code that uses a Producer and >handles connectivity errors to Kafka brokers and/or ZooKeeper. >

Re: Kafka Mock

I will have a look. Thanks mike marzo 908 209-4484 On Apr 30, 2014 4:56 PM, "Olson,Andrew" wrote: > You might take a look at the KafkaServerTestHarness [1] if you’re not > already familiar with it. It’s not exactly “mock”, but it makes it easy to > test Kafka functionality without needing to co

Re: Kafka Mock

You might take a look at the KafkaServerTestHarness [1] if you’re not already familiar with it. It’s not exactly “mock”, but it makes it easy to test Kafka functionality without needing to connect to any external broker or ZK processes. We ported this test harness to Java for our junit and integ

RE: 0.8.1 cpu usage

Please forget the cause i found in my previous email. it is not true. i will keep investigating this issue. From: yu_l...@hotmail.com To: users@kafka.apache.org Subject: 0.8.1 cpu usage Date: Wed, 30 Apr 2014 11:10:07 -0400 Hi team, We have noticed that the cpu usage of 0.8.1 has more than

Re: How to replace kafka.cluster.Broker.getZkString() ?

Alex, We removed getZkString(). You can use getConnectionString() instead. Thanks, Neha On Wed, Apr 30, 2014 at 8:32 AM, Guozhang Wang wrote: > Hi Alex, > > I cannot find any callers to Broker.getZkString() in trunk or 0.8.1.1, > could you point out for me? > > Guozhang > > > On Wed, Apr 30,

Re: Kafka Mock

Matt - Here's a small Gist that shows some of the setup for Consumer.createJavaConsumer(). https://gist.github.com/clarkbreyman/548f25ad78c4bb4a7df7 -Clark On Wed, Apr 30, 2014 at 7:33 AM, Matt Narrell wrote: > Clark, > > Would you attach a small example of this mocking of > Consumer.createJava

Re: Kafka Mock

With the new Java Kafka clients we are shipping a mock producer and consumer. E.g. https://github.com/apache/kafka/blob/0.8.1/clients/src/main/java/org/apache/kafka/clients/producer/MockProducer.java I suppose that doesn't help you now unless you are using the new producer already, but at least we

Re: 0.8.1 cpu usage

Hm, this seems counter-intuitive. Running 3x per second vs 2x per second should not really register on a modern CPU, right? Can you try this and see if you notice any difference? On Wed, Apr 30, 2014 at 8:10 AM, Libo Yu wrote: > Hi team, > > We have noticed that the cpu usage of 0.8.1 has more

Re: How to replace kafka.cluster.Broker.getZkString() ?

Hi Alex, I cannot find any callers to Broker.getZkString() in trunk or 0.8.1.1, could you point out for me? Guozhang On Wed, Apr 30, 2014 at 7:16 AM, Alex The Rocker wrote: > Hello, > > > > I am currently trying to upgrade from our system working with Kafka 0.8 > beta1 to Kafka 0.8.1.1, and I

Re: Brokers throwing warning messages after change in retention policy and multiple produce failures

Let us know if you still see this issue in 0.8.1.1. On Mon, Apr 28, 2014 at 3:35 PM, Sadhan Sood wrote: > Thank you @Guozhang and @Drew for your responses. We don't see any errors > in server logs but not sure if we could've run into KAFKA-1311. We are > using 0.8.0 on brokers and 0.8.1 on pro

0.8.1 cpu usage

Hi team, We have noticed that the cpu usage of 0.8.1 has more than doubled compared to beta1. I captured a thread dump and compared source code. I believe the higher cpu usage is introduced by the change of timeout value used by selector.select() in SocketServer.scala.In beta1 it is 500ms. In 0

Re: Kafka Mock

Clark, Would you attach a small example of this mocking of Consumer.createJavaConsumer please? Thanks, Matt On Apr 30, 2014, at 6:39 AM, Clark Breyman wrote: > Mike, > > It's nowhere near a full broker, but I've had luck with using PowerMock > (Mockito version) since it can mock out the > st

How to replace kafka.cluster.Broker.getZkString() ?

Hello, I am currently trying to upgrade from our system working with Kafka 0.8 beta1 to Kafka 0.8.1.1, and I noticed that in some of our code we now get a MethodNotFound exception, caused by the removal of kafka.cluster.Broker.getZkString() method. Would somebody know what is the recommended

Re: Kafka metrics: Bytesout greater than Bytesin??

Jun, Thanks for the pointer to the docs. It would be great if more detail like this discussion were included in the Wiki. The detail of each metric is not always clear. For our case (trying to monitor and alert on if the consumer is falling far behind the producer on a specific topic), what would

Re: Kafka Mock

Mike, It's nowhere near a full broker, but I've had luck with using PowerMock (Mockito version) since it can mock out the static Consumer.createJavaConsumerConnector and the java Producer constructor. It doesn't guarantee that your mocks behave like Kafka but it's something. If you find/create som

Need example of robust Producer

I am new to Kafka. I'd like to see example code that uses a Producer and handles connectivity errors to Kafka brokers and/or ZooKeeper. I have seen some code that creates and tears down a Producer for each message that is sent, but this doesn't seem right to me. The same example code, queries Z