The maintainer of librdkafka was able to reproduce the latency. He thinks
it may be some sort of batching algorithm similar to Nagle inside OpenSSL.
Status of the issue is maintained at:
https://github.com/edenhill/librdkafka/issues/920
Thanks to all on this mailing list for your help in diagnos
So the kafka performance tools seem to indicate that the problem is not in
the broker, but rather somewhere in librdkafka/OpenSSL. I'm not completely
sure I got the configs right to try and eliminate any batching
considerations in the latency calculation (it seems like encrypting /
decrypting a ba
Thank you both, Hans and Rajini.
I will try out all the methods you suggested and report back.
As an aside my investigation into the known, slow software implementation
of the GCM class of cipher algorithms in java 8 was a bust. I tried all of
the default cipher suites common to OpenSSL (on the
You can use the tools shipped with Kafka to measure latency.
For latency at low load, run:
- bin/kafka-run-class.sh kafka.tools.EndToEndLatency
You may also find it useful to run producer performance test at different
throughputs. The tool prints out latency as well:
- bin/kafka-produc
Publish lots of messages and measure in seconds or minutes. Otherwise you are
just benchmarking the initial SSL handshake setup time which should normally be
a one time overhead, not a per message overhead. If you just send one message
then of course SSL is much slower.
-hans
> On Nov 18, 2016
Hi, Hans. I was able to get the command line producer / consumer working
with SSL but I'm not sure how to measure millisecond resolution latency
with them. I thought maybe the '--property print.timestamp=true' argument
would help, but only has second resolution. Do you know of any way to get
th
What is the difference using the bin/kafka-console-producer and
kafka-console-consumer as pub/sub clients?
see http://docs.confluent.io/3.1.0/kafka/ssl.html
-hans
/**
* Hans Jespersen, Principal Systems Engineer, Confluent Inc.
* h...@confluent.io (650)924-2670
*/
On Thu, Nov 17, 2016 at 11:
