You can use the tools shipped with Kafka to measure latency. For latency at low load, run:
- bin/kafka-run-class.sh kafka.tools.EndToEndLatency You may also find it useful to run producer performance test at different throughputs. The tool prints out latency as well: - bin/kafka-producer-perf-test.sh On Fri, Nov 18, 2016 at 1:25 AM, Hans Jespersen <h...@confluent.io> wrote: > Publish lots of messages and measure in seconds or minutes. Otherwise you > are just benchmarking the initial SSL handshake setup time which should > normally be a one time overhead, not a per message overhead. If you just > send one message then of course SSL is much slower. > > -hans > > > On Nov 18, 2016, at 1:07 AM, Aaron Wilkinson <aa...@modopayments.com> > wrote: > > > > Hi, Hans. I was able to get the command line producer / consumer working > > with SSL but I'm not sure how to measure millisecond resolution latency > > with them. I thought maybe the '--property print.timestamp=true' > argument > > would help, but only has second resolution. Do you know of any way to > get > > the consumer to print out a receipt time-stamp with millisecond > > resolution? Or of any extended documentation on the command line tools > in > > general? > > > > Oh also, a couple other tidbits that may help: > > Ubuntu 16.04 > > Kafka 10.1.0 > > openjdk version "1.8.0_111" > > TLS 1.2 > > > > I was wondering if maybe this could be my problem: > > http://stackoverflow.com/questions/25992131/slow-aes- > gcm-encryption-and-decryption-with-java-8u20 > > > > I didn't specify any cipher suites in either the broker or the client > > config which I gather leaves it up to the broker/client to decide during > > TLS handshaking. I'm not sure if there is an easy way to figure out > which > > one they ended up with... I'll work on specifying which cipher suite I > > want and try to pick something with which java is simpatico. > > > > > >> On Thu, Nov 17, 2016 at 4:04 PM, Hans Jespersen <h...@confluent.io> > wrote: > >> > >> What is the difference using the bin/kafka-console-producer and > >> kafka-console-consumer as pub/sub clients? > >> > >> see http://docs.confluent.io/3.1.0/kafka/ssl.html > >> > >> -hans > >> > >> /** > >> * Hans Jespersen, Principal Systems Engineer, Confluent Inc. > >> * h...@confluent.io (650)924-2670 > >> */ > >> > >> On Thu, Nov 17, 2016 at 11:56 PM, Aaron Wilkinson < > aa...@modopayments.com> > >> wrote: > >> > >>> Pardon if this is a oft repeated issue, but all the information I could > >>> find said I should expect a 20-50% performance hit when using SSL with > >>> kafka, and I am seeing closer to 2000-3000% > >>> > >>> I'm trying to get kafka to behave like a fast, secured message bus. > So I > >>> am sending small messages, one at a time. I have set up a simple, 2 > >>> machine experiment in AWS with 1 client machine and 1 zookeeper/broker > >>> machine and I'm an running a very linear test. > >>> > >>> There are 2 topics: "request" and "response" and 2 threads on the > client > >>> machine each of which connects to those 2 topics. Thread 1 produces a > >>> "request", thread 2 consumes it and then produces a "response" which > >> thread > >>> 1 then consumes. At that point thread 1 proceeds to send the next > >>> "request" and the process repeats. > >>> > >>> So there are a total of 4 connections to the broker. > >>> > >>> I can run a sustained test without SSL and see 1 to 1.5 ms per message > >> hop > >>> (where a "hop" means the message has traveled across 1 of the 4 > >>> connections- either a production or a consumption of either the request > >> or > >>> the response). > >>> > >>> Each connection for which I turn on SSL increases the hop time 35 to 45 > >> ms. > >>> > >>> Now, the problem could be with the stack I'm using (PHP 7 talking to > the > >>> broker via the librdkafka C library). But before I go about trying to > >>> reproduce this with a java client (which is not my forte) I was > wondering > >>> if anyone else has run into a similar issue either with PHP or any > other > >>> language / library. Or does anyone know a direct way to figure out > >> whether > >>> this slow down is at the broker or at the client? > >>> > >>> Thanks in advance for your help! > >>> Aaron > >>> > >> > -- Regards, Rajini
