Re: Perf producer/consumers for compacted topics

2016-05-18 Thread Tom Crayford
Hi, I'm interested in benchmarking the impact of compaction on producers and consumers and long term cluster stability. That's not *quite* the impact of it on the server side, but it certainly plays into it. For example, I'd like to be able to answer "in configuration X, if we write N messages int

Re: Perf producer/consumers for compacted topics

2016-05-18 Thread Manikumar Reddy
Hi, There is a kafka.tools.TestLogCleaning tool, which is used to stress test the compaction feature. This tool validates the correctness of compaction process. This tool can be improved for perf testing. I think you want to benchmark server side compaction process. Currently we have few compact

Perf producer/consumers for compacted topics

2016-05-17 Thread Tom Crayford
Hi there, As noted in the 0.10.0.0-RC4 release thread, we (Heroku Kafka) have been doing extensive benchmarking of Kafka. In our case this is to help give customers a good idea of the performance of our various configurations. For this we orchestrate the Kafka `producer-perf.sh` and `consumer-perf