Hi Todd, I agree that KAFKA-2561 would be good to have for the reasons you state.
Ismael On Mon, Mar 6, 2017 at 5:17 PM, Todd Palino <tpal...@gmail.com> wrote: > Thanks for the link, Ismael. I had thought that the most recent kernels > already implemented this, but I was probably confusing it with BSD. Most of > my systems are stuck in the stone age right now anyway. > > It would be nice to get KAFKA-2561 in, either way. First off, if you can > take advantage of it it’s a good performance boost. Second, especially with > the security landscape getting worse and worse, it would be good to have > options as far as the TLS implementation goes. A zero-day exploit in the > Java TLS implementation would be devastating, and more difficult to react > to as it would require a new JRE (bringing with it who knows what > problems). Swapping an underlying OpenSSL version would be much more > palatable. > > -Todd > > > On Mon, Mar 6, 2017 at 9:01 AM, Ismael Juma <ism...@juma.me.uk> wrote: > > > Even though OpenSSL is much faster than the Java 8 TLS implementation (I > > haven't tested against Java 9, which is much faster than Java 8, but > > probably still slower than OpenSSL), all the tests were without zero copy > > in the sense that is being discussed here (i.e. sendfile). To benefit > from > > sendfile with TLS, kernel-level changes/modules are required: > > > > https://github.com/ktls/af_ktls > > http://www.phoronix.com/scan.php?page=news_item&px=FreeBSD- > Faster-Sendfile > > > > Ismael > > > > On Mon, Mar 6, 2017 at 4:18 PM, Todd Palino <tpal...@gmail.com> wrote: > > > > > So that’s not quite true, Hans. First, as far as the performance hit > > being > > > not a big impact (25% is huge). Or that it’s to be expected. Part of > the > > > problem is that the Java TLS implementation does not support zero copy. > > > OpenSSL does, and in fact there’s been a ticket open to allow Kafka to > > > support using OpenSSL for a while now: > > > > > > https://issues.apache.org/jira/browse/KAFKA-2561 > > > > > > > > > > > > > > > On Mon, Mar 6, 2017 at 6:30 AM, Hans Jespersen <h...@confluent.io> > > wrote: > > > > > > > > > > > Its not a single message at a time that is encrypted with TLS its the > > > > entire network byte stream so a Kafka broker can’t even see the Kafka > > > > Protocol tunneled inside TLS unless it’s terminated at the broker. > > > > It is true that losing the zero copy optimization impacts performance > > > > somewhat but it’s not what I would call a “big impact” because Kafka > > > does > > > > a lot of other things to get it’s performance (like using page cache > > and > > > > doing lots on sequential disk I/O). The difference should be > something > > in > > > > the order of 25-30% slower with TLS enabled which is about what you > > would > > > > see with any other messaging protocol with TLS on vs off. > > > > > > > > If you wanted to encrypt each message independently before sending to > > > > Kafka then zero copy would still be in effect and all the consumers > > would > > > > get the same encrypted message (and have to understand how to decrypt > > > it). > > > > > > > > -hans > > > > > > > > > > > > > > > > > On Mar 6, 2017, at 5:38 AM, Nicolas Motte <lingusi...@gmail.com> > > > wrote: > > > > > > > > > > Hi everyone, > > > > > > > > > > I understand one of the reasons why Kafka is performant is by using > > > > > zero-copy. > > > > > > > > > > I often hear that when encryption is enabled, then Kafka has to > copy > > > the > > > > > data in user space to decode the message, so it has a big impact on > > > > > performance. > > > > > > > > > > If it is true, I don t get why the message has to be decoded by > > Kafka. > > > I > > > > > would assume that whether the message is encrypted or not, Kafka > > simply > > > > > receives it, appends it to the file, and when a consumer wants to > > read > > > > it, > > > > > it simply reads at the right offset... > > > > > > > > > > Also I m wondering if it s the case if we don t use keys (pure > > queuing > > > > > system with key=null). > > > > > > > > > > Cheers > > > > > Nico > > > > > > > > > > > > > > > > > -- > > > *Todd Palino* > > > Staff Site Reliability Engineer > > > Data Infrastructure Streaming > > > > > > > > > > > > linkedin.com/in/toddpalino > > > > > > > > > -- > *Todd Palino* > Staff Site Reliability Engineer > Data Infrastructure Streaming > > > > linkedin.com/in/toddpalino >
