I'm hesitant to cite it because it wasn't really a proper benchmark, but with the end-to-end encryption through Kafka proof of concept described at http://symc.ly/1pC2CEG, doing the encryption added only 26% to the time taken to send messages and only 6% to the time taken to consume messages. This is with batching 200 300-byte messages together for encryption. More details are in the blog post.
Personally I think that encrypting sensitive data before handing it to Kafka (or at least before it leaves the producing box) just makes sense to do if the situation allows. The Kafka installation wouldn't be able reveal the data even if its systems and networks are compromised because it never sees the data in the clear and doesn't know how to decrypt it. In the way we set it up, someone would need the recipient's RSA private key to decrypt (or would need to have compromised a decrypting system). -- Jim On 6/2/16, 2:56 AM, "Tom Crayford" <tcrayf...@heroku.com> wrote: >Filesystem encryption is transparent to Kafka. You don't need to use SSL, >but your encryption requirements may cause you to need SSL as well. > >With regards to compression, without adding at rest encryption to Kafka >(which is a very major piece of work, one that for sure requires a KIP and >has many, many implications), there's not much to do there. I think it's >worth examining your threat models that require encryption on disk without >full disk encryption being suitable. Generally compromised broker machines >means an attacker will be able to sniff in flight traffic anyway, if the >goal is to never leak messages even if an attacker has full control of the >broker machine, I'd suggest that that seems pretty impossible under >current >operating environments. > >If the issue is compliance, I'd recommend querying whichever compliance >standard you're operating under about the suitability of full disk >encryption, and careful thought about encrypting the most sensitive parts >of messages. Whilst encryption in the producer and consumer does lead to >performance issues and decrease the capability of compression to shrink a >dataset, doing partial encryption of messages is easy enough. > >Generally we've found that the kinds of uses of Kafka that require in >message encryption (alongside full disk encryption and SSL which we >provide >as standard) don't have such high throughput needs that they worry about >compression etc. That clearly isn't true for all use cases though. > >Thanks > >Tom Crayford >Heroku Kafka > >On Thursday, 2 June 2016, Gerard Klijs <gerard.kl...@dizzit.com> wrote: > >> You could add a header to every message, with information whether it's >> encrypted or not, then you don't have to encrypt all the messages, or >>you >> only do it for some topics. >> >> On Thu, Jun 2, 2016 at 6:36 AM Bruno Rassaerts < >> bruno.rassae...@novazone.be <javascript:;>> >> wrote: >> >> > It works indeed but encrypting individual messages really influences >>the >> > batch compression done by Kafka. >> > Performance drops to about 1/3 of what it is without (even if we >>prepare >> > the encrypted samples upfront). >> > In the end what we going for is only encrypting what we really really >> need >> > to encrypt, not every message systematically. >> > >> > > On 31 May 2016, at 13:00, Gerard Klijs <gerard.kl...@dizzit.com >> <javascript:;>> wrote: >> > > >> > > If you want system administrators not being able to see the data, >>the >> > only >> > > option is encryption, with only the clients sharing the key (or >> whatever >> > is >> > > used to (de)crypt the data). Like the example from eugene. I don't >>know >> > the >> > > kind of messages you have, but you could always wrap something >>around >> any >> > > (de)serializer your currently using. >> > > >> > > On Tue, May 31, 2016 at 12:21 PM Bruno Rassaerts < >> > > bruno.rassae...@novazone.be <javascript:;>> wrote: >> > > >> > >> I’ve asked the same question in the past, and disk encryption was >> > >> suggested as a solution as well. >> > >> However, as far as I know, disk encryption will not prevent your >>data >> to >> > >> be stolen when the machine is compromised. >> > >> What we are looking for is even an additional barrier, so that even >> > system >> > >> administrators do not have access to the data. >> > >> Any suggestions ? >> > >> >> > >>> On 24 May 2016, at 14:40, Tom Crayford <tcrayf...@heroku.com >> <javascript:;>> wrote: >> > >>> >> > >>> Hi, >> > >>> >> > >>> There's no encryption at rest. It's recommended to use filesystem >> > >>> encryption, or encryption of each individual message before >>producing >> > it >> > >>> for this. >> > >>> >> > >>> Only the new producer and consumers have SSL support. >> > >>> >> > >>> Thanks >> > >>> >> > >>> Tom Crayford >> > >>> Heroku Kafka >> > >>> >> > >>> On Tue, May 24, 2016 at 11:33 AM, Snehalata Nagaje < >> > >>> snehalata.nag...@harbingergroup.com <javascript:;>> wrote: >> > >>> >> > >>>> >> > >>>> >> > >>>> Thanks for quick reply. >> > >>>> >> > >>>> Do you mean If I see messages in kafka, those will not be >>readable? >> > >>>> >> > >>>> And also, we are using new producer but old consumer , does old >> > consumer >> > >>>> have ssl support? >> > >>>> >> > >>>> As mentioned in document, its not there. >> > >>>> >> > >>>> >> > >>>> Thanks, >> > >>>> Snehalata >> > >>>> >> > >>>> ----- Original Message ----- >> > >>>> From: "Mudit Kumar" <mudit.ku...@askme.in <javascript:;>> >> > >>>> To: users@kafka.apache.org <javascript:;> >> > >>>> Sent: Tuesday, May 24, 2016 3:53:26 PM >> > >>>> Subject: Re: Kafka encryption >> > >>>> >> > >>>> Yes,it does that.What specifically you are looking for? >> > >>>> >> > >>>> >> > >>>> >> > >>>> >> > >>>> On 5/24/16, 3:52 PM, "Snehalata Nagaje" < >> > >>>> snehalata.nag...@harbingergroup.com <javascript:;>> wrote: >> > >>>> >> > >>>>> Hi All, >> > >>>>> >> > >>>>> >> > >>>>> We have requirement of encryption in kafka. >> > >>>>> >> > >>>>> As per docs, we can configure kafka with ssl, for secured >> > >> communication. >> > >>>>> >> > >>>>> But does kafka also stores data in encrypted format? >> > >>>>> >> > >>>>> >> > >>>>> Thanks, >> > >>>>> Snehalata >> > >>>> >> > >> >> > >> >> > >> > >>
