Its not a single message at a time that is encrypted with TLS its the entire network byte stream so a Kafka broker can’t even see the Kafka Protocol tunneled inside TLS unless it’s terminated at the broker. It is true that losing the zero copy optimization impacts performance somewhat but it’s not what I would call a “big impact” because Kafka does a lot of other things to get it’s performance (like using page cache and doing lots on sequential disk I/O). The difference should be something in the order of 25-30% slower with TLS enabled which is about what you would see with any other messaging protocol with TLS on vs off.
If you wanted to encrypt each message independently before sending to Kafka then zero copy would still be in effect and all the consumers would get the same encrypted message (and have to understand how to decrypt it). -hans > On Mar 6, 2017, at 5:38 AM, Nicolas Motte <lingusi...@gmail.com> wrote: > > Hi everyone, > > I understand one of the reasons why Kafka is performant is by using > zero-copy. > > I often hear that when encryption is enabled, then Kafka has to copy the > data in user space to decode the message, so it has a big impact on > performance. > > If it is true, I don t get why the message has to be decoded by Kafka. I > would assume that whether the message is encrypted or not, Kafka simply > receives it, appends it to the file, and when a consumer wants to read it, > it simply reads at the right offset... > > Also I m wondering if it s the case if we don t use keys (pure queuing > system with key=null). > > Cheers > Nico
