Pardon if this is a oft repeated issue, but all the information I could find said I should expect a 20-50% performance hit when using SSL with kafka, and I am seeing closer to 2000-3000%
I'm trying to get kafka to behave like a fast, secured message bus. So I am sending small messages, one at a time. I have set up a simple, 2 machine experiment in AWS with 1 client machine and 1 zookeeper/broker machine and I'm an running a very linear test. There are 2 topics: "request" and "response" and 2 threads on the client machine each of which connects to those 2 topics. Thread 1 produces a "request", thread 2 consumes it and then produces a "response" which thread 1 then consumes. At that point thread 1 proceeds to send the next "request" and the process repeats. So there are a total of 4 connections to the broker. I can run a sustained test without SSL and see 1 to 1.5 ms per message hop (where a "hop" means the message has traveled across 1 of the 4 connections- either a production or a consumption of either the request or the response). Each connection for which I turn on SSL increases the hop time 35 to 45 ms. Now, the problem could be with the stack I'm using (PHP 7 talking to the broker via the librdkafka C library). But before I go about trying to reproduce this with a java client (which is not my forte) I was wondering if anyone else has run into a similar issue either with PHP or any other language / library. Or does anyone know a direct way to figure out whether this slow down is at the broker or at the client? Thanks in advance for your help! Aaron
