why kafka producer api use cpu so high?

I write a very simple code , like this : public class LogProducer { private Producer inner; public LogProducer() throws Exception{ Properties properties = new Properties(); properties.load(ClassLoader.getSystemResourceAsStream("producer.properties")); Pr

Re: question about mirror maker

Well, if you have a cluster in each datacenter, all with the same topics, you can¹t just mirror the messages between them, as you will create a loop. The way we do it is to have a ³local² cluster and an ³aggregate² cluster. The local cluster has the data for only that datacenter. Then we run mirror

Re: why kafka producer api use cpu so high?

What is your compression configuration for your producer? One of the biggest CPU source for the producer is doing compression and also checksuming. Tim On Sun, May 11, 2014 at 12:24 AM, wrote: > I write a very simple code , like this : > public class LogProducer { > > private Producer inne

Re: why kafka producer api use cpu so high?

This code says to send this message infinitely as fast as the machine can thereby consuming as much of one CPU as possible. You may want to consider an alternate test, perhaps one that records the number of messages sent in a given time period. > > public static void main(String[] args) { > >

Re: Re: why kafka producer api use cpu so high?

because my app can generate 50MB log every second and one record of log is about 1KB , so I must send this log as fast as machine can. this is very difficult, on one hand I want to send log as fast as possible, on the other hand I want kafka producer api use cpu as low as possible. if kafk

Re: Re: why kafka producer api use cpu so high?

I use snappy for compression. but even without compression, this procedure also use 50% one core cpu. when using snappy ,this procedure use 100% one core cpu. From: Timothy Chen Date: 2014-05-11 15:53 To: users@kafka.apache.org Subject: Re: why kafka producer api use cpu so high? What is your

Re: 0.7.1 Will there be any log info under poor network which packet may lose?

Jay, Thanks for the reply ;-) That helps. On Tue, May 6, 2014 at 1:38 AM, Jay Kreps wrote: > TCP will attempt to resend until the packets are successfully delivered or > a timeout occurs. So a packet loss should not lead to an error. The error > you see in your log is the queue of unsent event

Re: question about mirror maker

Hi Todd, Thanks for your answer. with regard to fail over for mirror maker, does that mean if i have 4 mirror maker running in different machines with same consumer group, it will auto load balance if one of the mirror maker fails ? Also, it looks to prevent mirror maker commit wrong (consumer wor

Re: Re: why kafka producer api use cpu so high?

If a process is CPU bound (which this producer almost certainly will be), it's going to consume as much CPU as it can to do what what it does. The test is flawed. Because there's no end state, the while loop is just going to burn CPU and, because it's singly threaded, it will take a single core. A

New consumer APIs

All: I've been going over the new consumer APIs and it seems like we're squishing a lot of different concerns together into a single class. The scope of the new Consumer is kind of all over the place. Managing the lifecycle - and especially the thread safety - seems challenging. Specifically, Cons

kafka dynamic topic discovery

Hi Guozhang Wang & Neha Narkhede , Thanks for your answers for this question http://qnalist.com/questions/4846254/kafka-dynamic-topic-discovery .I tried high level consumer in the following example https://cwiki.apac

Re: question about mirror maker

Yes, on both counts. Putting the mirror maker in the same datacenter in the target cluster is exactly what we do as well. We also monitor both the consumer lag (by comparing the offsets stored in Zookeeper and the tail offset on the brokers), and the number of dropped and failed messages on the mir

Re: question about isr

It seems that broker 1 had a soft failure (potentially due to GC). However, somehow it didn't receive the latest LeaderAndIsrRequest from the controller. Which version of Kafka are you using? In the controller log, do you see broker 1 being added back as a live broker? Thanks, Jun On Fri, May 9

Re: question about isr

Jun - We're using 0.8.1 The timestamps in the last few entries in controller.log seem to correspond to when the trouble started in server.log controller.log: [2014-05-08 19:01:27,693] INFO [SessionExpirationListener on 1], ZK expired; shut down all controller components and try to re-elect (kafka

Re: kafka dynamic topic discovery

You can take a look at the source code of MirrorMaker.scala, which uses the wildcard api createMessageStreamsByFilter(). Thanks, Jun On Wed, May 7, 2014 at 6:05 AM, Kannadhasan Sekar wrote: > Hi Guozhang Wang & Neha > Narkhede