Re: Producer becomes slow over time

2015-09-29 Thread Helleren, Erik
Well, its possible that it is a networking issue which is why I asked for results from a test from a machine that isn¹t on the kafka cluster. And were all the producers run on the same member of the kafka cluster, or was it spread out? Also, for the topic you are testing with, how many partitions

Re: Producer becomes slow over time

2015-09-29 Thread Prabhjot Bharaj
Hi Erik, I've not tested it on a producer that is not a part of the kafka cluster. *Producer configuration on all machines:-* root@x.x.x.x:/a/kafka/config# cat producer.properties | egrep -v '^#|^$' metadata.broker.list=localhost:9092 producer.type=sync compression.codec=none serializer.cl

Re: Producer becomes slow over time

2015-09-24 Thread Helleren, Erik
What happens when the new producer that is getting 70 MB/s is started on a machine that is not part of the kafka cluster? Can you include your topic description/configuration, producer configuration, and broker configuration? On 9/24/15, 1:44 AM, "Prabhjot Bharaj" wrote: >Hi, > >I would like to

Re: Producer becomes slow over time

2015-09-23 Thread Prabhjot Bharaj
Hi, I would like to dig deep into this issue. I've changed log4j.properties for logging in ALL mode in all places. I am getting lost in the logs. Any pointers would be welcome Please let me know if you would need any information regarding this Thanks, Prabhjot On Wed, Sep 23, 2015 at 6:46 PM,

Producer becomes slow over time

2015-09-23 Thread Prabhjot Bharaj
Hello Folks, I've noticed that 2 producer machines, that I had configured, have become very slow over time They are giving 17-19 MB/s But, a producer that I setup today is giving 70MB/s as the write throughput If I see the contents of bin, libs, config directories, nothing is different in the fi