MG>quick questions for bruno and jim
> Subject: Re: Kafka encryption > From: bruno.rassae...@novazone.be > Date: Mon, 6 Jun 2016 10:51:13 +0200 > CC: tcrayf...@heroku.com > To: users@kafka.apache.org > > Indeed to get proper performance, messages need to be batched before > encryption. > However, this is not that straightforward to implement and Kafka has already > a very good batching algorithm. > For example, when do you decide to no longer wait for additional messages and > send a non-full batch ? Not that obvious. MG>Bruno can you describe kafka criteria for a Full Batch?MG>can you elucidate kafka criteria for a non Full-Batch? > > Ideally, we would like kafka clients to encrypt/decrypt the compressed > batches of kafka. > That seems like the ideal place to do this. > > > > On 03 Jun 2016, at 07:27, Jim Hoagland <jim_hoagl...@symantec.com> wrote: > > > > I'm hesitant to cite it because it wasn't really a proper benchmark, but > > with the end-to-end encryption through Kafka proof of concept described at > > http://symc.ly/1pC2CEG, doing the encryption added only 26% to the time > > taken to send messages and only 6% to the time taken to consume messages. > > This is with batching 200 300-byte messages together for encryption. More > > details are in the blog post. > > > > Personally I think that encrypting sensitive data before handing it to > > Kafka (or at least before it leaves the producing box) just makes sense to > > do if the situation allows. The Kafka installation wouldn't be able > > reveal the data even if its systems and networks are compromised because > > it never sees the data in the clear and doesn't know how to decrypt it. > > In the way we set it up, someone would need the recipient's RSA private > > key to decrypt (or would need to have compromised a decrypting system). MG>Jim can we assume you only implement Asymmetric Cryptography? > > > > -- Jim > > > > > > On 6/2/16, 2:56 AM, "Tom Crayford" <tcrayf...@heroku.com> wrote: > > > >> Filesystem encryption is transparent to Kafka. You don't need to use SSL, > >> but your encryption requirements may cause you to need SSL as well. > >> > >> With regards to compression, without adding at rest encryption to Kafka > >> (which is a very major piece of work, one that for sure requires a KIP and > >> has many, many implications), there's not much to do there. I think it's > >> worth examining your threat models that require encryption on disk without > >> full disk encryption being suitable. Generally compromised broker machines > >> means an attacker will be able to sniff in flight traffic anyway, if the > >> goal is to never leak messages even if an attacker has full control of the > >> broker machine, I'd suggest that that seems pretty impossible under > >> current > >> operating environments. > >> > >> If the issue is compliance, I'd recommend querying whichever compliance > >> standard you're operating under about the suitability of full disk > >> encryption, and careful thought about encrypting the most sensitive parts > >> of messages. Whilst encryption in the producer and consumer does lead to > >> performance issues and decrease the capability of compression to shrink a > >> dataset, doing partial encryption of messages is easy enough. > >> > >> Generally we've found that the kinds of uses of Kafka that require in > >> message encryption (alongside full disk encryption and SSL which we > >> provide > >> as standard) don't have such high throughput needs that they worry about > >> compression etc. That clearly isn't true for all use cases though. > >> > >> Thanks > >> > >> Tom Crayford > >> Heroku Kafka > >> > >> On Thursday, 2 June 2016, Gerard Klijs <gerard.kl...@dizzit.com> wrote: > >> > >>> You could add a header to every message, with information whether it's > >>> encrypted or not, then you don't have to encrypt all the messages, or > >>> you > >>> only do it for some topics. > >>> > >>> On Thu, Jun 2, 2016 at 6:36 AM Bruno Rassaerts < > >>> bruno.rassae...@novazone.be <javascript:;>> > >>> wrote: > >>> > >>>> It works indeed but encrypting individual messages really influences > >>> the > >>>> batch compression done by Kafka. > >>>> Performance drops to about 1/3 of what it is without (even if we > >>> prepare > >>>> the encrypted samples upfront). > >>>> In the end what we going for is only encrypting what we really really > >>> need > >>>> to encrypt, not every message systematically. > >>>> > >>>>> On 31 May 2016, at 13:00, Gerard Klijs <gerard.kl...@dizzit.com > >>> <javascript:;>> wrote: > >>>>> > >>>>> If you want system administrators not being able to see the data, > >>> the > >>>> only > >>>>> option is encryption, with only the clients sharing the key (or > >>> whatever > >>>> is > >>>>> used to (de)crypt the data). Like the example from eugene. I don't > >>> know > >>>> the > >>>>> kind of messages you have, but you could always wrap something > >>> around > >>> any > >>>>> (de)serializer your currently using. > >>>>> > >>>>> On Tue, May 31, 2016 at 12:21 PM Bruno Rassaerts < > >>>>> bruno.rassae...@novazone.be <javascript:;>> wrote: > >>>>> > >>>>>> I’ve asked the same question in the past, and disk encryption was > >>>>>> suggested as a solution as well. > >>>>>> However, as far as I know, disk encryption will not prevent your > >>> data > >>> to > >>>>>> be stolen when the machine is compromised. > >>>>>> What we are looking for is even an additional barrier, so that even > >>>> system > >>>>>> administrators do not have access to the data. > >>>>>> Any suggestions ? > >>>>>> > >>>>>>> On 24 May 2016, at 14:40, Tom Crayford <tcrayf...@heroku.com > >>> <javascript:;>> wrote: > >>>>>>> > >>>>>>> Hi, > >>>>>>> > >>>>>>> There's no encryption at rest. It's recommended to use filesystem > >>>>>>> encryption, or encryption of each individual message before > >>> producing > >>>> it > >>>>>>> for this. > >>>>>>> > >>>>>>> Only the new producer and consumers have SSL support. > >>>>>>> > >>>>>>> Thanks > >>>>>>> > >>>>>>> Tom Crayford > >>>>>>> Heroku Kafka > >>>>>>> > >>>>>>> On Tue, May 24, 2016 at 11:33 AM, Snehalata Nagaje < > >>>>>>> snehalata.nag...@harbingergroup.com <javascript:;>> wrote: > >>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> Thanks for quick reply. > >>>>>>>> > >>>>>>>> Do you mean If I see messages in kafka, those will not be > >>> readable? > >>>>>>>> > >>>>>>>> And also, we are using new producer but old consumer , does old > >>>> consumer > >>>>>>>> have ssl support? > >>>>>>>> > >>>>>>>> As mentioned in document, its not there. > >>>>>>>> > >>>>>>>> > >>>>>>>> Thanks, > >>>>>>>> Snehalata > >>>>>>>> > >>>>>>>> ----- Original Message ----- > >>>>>>>> From: "Mudit Kumar" <mudit.ku...@askme.in <javascript:;>> > >>>>>>>> To: users@kafka.apache.org <javascript:;> > >>>>>>>> Sent: Tuesday, May 24, 2016 3:53:26 PM > >>>>>>>> Subject: Re: Kafka encryption > >>>>>>>> > >>>>>>>> Yes,it does that.What specifically you are looking for? > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> On 5/24/16, 3:52 PM, "Snehalata Nagaje" < > >>>>>>>> snehalata.nag...@harbingergroup.com <javascript:;>> wrote: > >>>>>>>> > >>>>>>>>> Hi All, > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> We have requirement of encryption in kafka. > >>>>>>>>> > >>>>>>>>> As per docs, we can configure kafka with ssl, for secured > >>>>>> communication. > >>>>>>>>> > >>>>>>>>> But does kafka also stores data in encrypted format? > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> Thanks, > >>>>>>>>> Snehalata > >>>>>>>> > >>>>>> > >>>>>> > >>>> > >>>> > >>> > > >
