Thanks, Otis, I'll review the info there. Also, after posting the original message, I came across some kafka extensions to popular industry (OSS) load test tools:
https://github.com/BrightTag/kafkameter https://github.com/mnogu/gatling-kafka they might be useful for tests that require customizations (rather than standard benchmark tests). On Sun, Sep 20, 2015 at 7:02 AM, Otis Gospodnetić < otis.gospodne...@gmail.com> wrote: > Hi, > > A few pointers are in this Kafka user ML thread: > http://search-hadoop.com/m/uyzND177HP92xnm4e > > Otis > -- > Monitoring * Alerting * Anomaly Detection * Centralized Log Management > Solr & Elasticsearch Support * http://sematext.com/ > > > On Sun, Sep 20, 2015 at 1:14 AM, David Luu <manga...@gmail.com> wrote: > > > I'd like to generate load against a system we have that uses kafka as the > > message bus. We have a custom JSON message format, and to properly load > > test the system, each set of messages for a particular scenario (i.e. > user) > > needs to have a unique identifier, which it normally does. > > > > I think of using record & playback technique to capture messages that > > correspond to a few users. Then play back those messages to generate load > > but to be realistic simulation and to scale up the load, I would: > > > > * re-use the captured user set to simulate additional users to scale up # > > of users against the system > > > > * for original set of users and when scaling beyond that for more users, > I > > would dynamically replace the identifier in the captured messages with a > > unique one generated at runtime for each user. also replacing anything > else > > that needs to like timestamps. > > > > As such, this would have to be a scripted solution. I don't think there > is > > existing kafka-centric tool to assist with such testing is there? > > > > If not, I'd likely have to build my own. In which case, my question is > what > > technology stack to use would be most suitable so that I can generate the > > highest amount of load with the least amount of load generating > > machines/hardware. Using threads, processes, or whatever. node.js, > python, > > scala, ruby, java, .net, etc. > > > > thoughts, suggestions appreciated, > > David > > >
