What is your compression configuration for your producer?
One of the biggest CPU source for the producer is doing compression
and also checksuming.
Tim
On Sun, May 11, 2014 at 12:24 AM, <yunbinw...@travelsky.com> wrote:
> I write a very simple code , like this :
> public class LogProducer {
>
> private Producer<String,String> inner;
> public LogProducer() throws Exception{
> Properties properties = new Properties();
>
> properties.load(ClassLoader.getSystemResourceAsStream("producer.properties"));
> ProducerConfig config = new ProducerConfig(properties);
> inner = new Producer<String, String>(config);
> }
>
>
> public void send(String topicName,String message) {
> if(topicName == null || message == null){
> return;
> }
> KeyedMessage<String, String> km = new KeyedMessage<String,
> String>(topicName,message);
> inner.send(km);
> }
> public void close(){
> inner.close();
> }
>
> /**
> * @param args
> */
> public static void main(String[] args) {
> LogProducer producer = null;
> try{
> producer = new LogProducer();
> int i=0;
> while(true){
> producer.send("test", "this is a
> sampleaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa");
> }
> }catch(Exception e){
> e.printStackTrace();
> }finally{
> if(producer != null){
> producer.close();
> }
> }
>
> }
>
> }
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> and the producer.properties like this:
> metadata.broker.list=127.0.0.1:9092
> producer.type=async
> serializer.class=kafka.serializer.StringEncoder
> batch.num.messages=200
> compression.codec=snappy
>
> I run this procedure on linux, which is 4 core cpu , 16GB memory.
> I find this procedure using one core cpu totally , this is "top" command
> ouput:
>
>
> [root@localhost ~]# top
> top - 13:51:09 up 5 days, 13:27, 3 users, load average: 0.96, 0.48, 0.35
> Tasks: 367 total, 3 running, 364 sleeping, 0 stopped, 0 zombie
> Cpu0 : 7.0%us, 0.3%sy, 0.0%ni, 92.0%id, 0.7%wa, 0.0%hi, 0.0%si, 0.0%st
> Cpu1 : 5.0%us, 0.0%sy, 0.0%ni, 95.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
> Cpu2 : 5.0%us, 0.0%sy, 0.0%ni, 95.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
> Cpu3 : 99.7%us, 0.3%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
> Mem: 16307528k total, 9398376k used, 6909152k free, 249952k buffers
> Swap: 8224760k total, 0k used, 8224760k free, 6071348k cached
>
> why producer api use cpu so high ? or maybe I make something wrong ?
>
> by the way , the kafka version 0.8.0 .
