Hi, There is a kafka.tools.TestLogCleaning tool, which is used to stress test the compaction feature. This tool validates the correctness of compaction process. This tool can be improved for perf testing.
I think you want to benchmark server side compaction process. Currently we have few compaction related metrics. We may need to add few more topic specific metrics for better analysis. log compaction related JMX metrics: kafka.log:type=LogCleaner,name=cleaner-recopy-percent kafka.log:type=LogCleaner,name=max-buffer-utilization-percent kafka.log:type=LogCleaner,name=max-clean-time-secs kafka.log:type=LogCleanerManager,name=max-dirty-percent Manikumar On Tue, May 17, 2016 at 8:45 PM, Tom Crayford <tcrayf...@heroku.com> wrote: > Hi there, > > As noted in the 0.10.0.0-RC4 release thread, we (Heroku Kafka) have been > doing extensive benchmarking of Kafka. In our case this is to help give > customers a good idea of the performance of our various configurations. For > this we orchestrate the Kafka `producer-perf.sh` and `consumer-perf.sh` > across multiple machines, which was relatively easy to do and very > successful (recently leading to a doc change and a good lesson about 0.10). > > However, we're finding one thing missing from the current producer/consumer > perf tests, which is that there's no good perf testing on compacted topics. > Some folk will undoubtedly use compacted topics, so it would be extremely > helpful (I think) for the community to have benchmarks that test > performance on compacted topics. We're interested in working on this and > contributing it upstream, but are pretty unsure what such a test should > look like. One straw proposal is to adapt the existing producer/consumer > perf tests to work on a compacted topic, likely with an additional flag on > the producer that lets you choose how wide a key range to emit, if it > should emit deletes (and how often to do so) and so on. Is there anything > more we could or should do there? > > We're happy writing the code here, and want to continue contributing back, > I'd just love a hand thinking about what perf tests for compacted topics > should look like. > > Thanks > > Tom Crayford > Heroku Kafka >
