Hi Todd Can you please help me with notes or document on how did you achieve encryption ?
I have followed data available on official sites but failed as I m no good with TLS . On Mar 6, 2017 19:55, "Todd Palino" <tpal...@gmail.com> wrote: > It’s not that Kafka has to decode it, it’s that it has to send it across > the network. This is specific to enabling TLS support (transport > encryption), and won’t affect any end-to-end encryption you do at the > client level. > > The operation in question is called “zero copy”. In order to send a message > batch to a consumer, the Kafka broker must read it from disk (sometimes > it’s cached in memory, but that’s irrelevant here) and send it across the > network. The Linux kernel allows this to happen without having to copy the > data in memory (to move it from the disk buffers to the network buffers). > However, if TLS is enabled, the broker must first encrypt the data going > across the network. This means that it can no longer take advantage of the > zero copy optimization as it has to make a copy in the process of applying > the TLS encryption. > > Now, how much of an impact this has on the broker operations is up for > debate, I think. Originally, when we ran into this problem was when TLS > support was added to Kafka and the zero copy send for plaintext > communications was accidentally removed as well. At the time, we saw a > significant performance hit, and the code was patched to put it back. > However, since then I’ve turned on inter-broker TLS in all of our clusters, > and when we did that there was no performance hit. This is odd, because the > replica fetchers should take advantage of the same zero copy optimization. > > It’s possible that it’s because it’s just one consumer (the replica > fetchers). We’re about to start testing additional consumers over TLS, so > we’ll see what happens at that point. All I can suggest right now is that > you test in your environment and see what the impact is. Oh, and using > message keys (or not) won’t matter here. > > -Todd > > > On Mon, Mar 6, 2017 at 5:38 AM, Nicolas Motte <lingusi...@gmail.com> > wrote: > > > Hi everyone, > > > > I understand one of the reasons why Kafka is performant is by using > > zero-copy. > > > > I often hear that when encryption is enabled, then Kafka has to copy the > > data in user space to decode the message, so it has a big impact on > > performance. > > > > If it is true, I don t get why the message has to be decoded by Kafka. I > > would assume that whether the message is encrypted or not, Kafka simply > > receives it, appends it to the file, and when a consumer wants to read > it, > > it simply reads at the right offset... > > > > Also I m wondering if it s the case if we don t use keys (pure queuing > > system with key=null). > > > > Cheers > > Nico > > > > > > -- > *Todd Palino* > Staff Site Reliability Engineer > Data Infrastructure Streaming > > > > linkedin.com/in/toddpalino >
