> Is this a case where multiple logical messages (when combined together) >are > treated by Kafka as a single message, and it's up to the consumer to > separate them?
Yes. -- Jim On 6/6/16, 7:12 AM, "Tom Brown" <tombrow...@gmail.com> wrote: >How would it be possible to encrypt an entire batch? My understanding >is that the Kafka server needs to know the boundaries of each message. >(E.g. The server decompresses compressed message sets and re-compresses >individual messages). > >Given that precedent, how could the server properly separate the >individual >messages without the server knowing the encryption keys or scheme? > >Is this a case where multiple logical messages (when combined together) >are >treated by Kafka as a single message, and it's up to the consumer to >separate them? > >--Tom > >On Monday, June 6, 2016, Jim Hoagland <jim_hoagl...@symantec.com> wrote: > >> MG>Jim can we assume you only implement Asymmetric Cryptography? >> >> >> As described and depicted in the blog post, we used asymmetric >> cryptography as the basis for trust, with symmetric crypto doing the >>heavy >> lifting. Specifically, for each "envelope", we include a randomly >> generated AES key encrypted with an RSA private key and the messages >> themselves are encrypted using the AES key. This is a common pattern. >> RSA should not be used to encrypt anything that is not random since >> otherwise patterns will be apparent. Also, RSA is more CPU intensive >>than >> AES. >> >> -- Jim >> >> On 6/6/16, 6:17 AM, "Martin Gainty" <mgai...@hotmail.com <javascript:;>> >> wrote: >> >> >MG>quick questions for bruno and jim >> > >> >> Subject: Re: Kafka encryption >> >> From: bruno.rassae...@novazone.be <javascript:;> >> >> Date: Mon, 6 Jun 2016 10:51:13 +0200 >> >> CC: tcrayf...@heroku.com <javascript:;> >> >> To: users@kafka.apache.org <javascript:;> >> >> >> >> Indeed to get proper performance, messages need to be batched before >> >>encryption. >> >> However, this is not that straightforward to implement and Kafka has >> >>already a very good batching algorithm. >> >> For example, when do you decide to no longer wait for additional >> >>messages and send a non-full batch ? Not that obvious. >> >MG>Bruno can you describe kafka criteria for a Full Batch?MG>can you >> >elucidate kafka criteria for a non Full-Batch? >> >> >> >> Ideally, we would like kafka clients to encrypt/decrypt the >>compressed >> >>batches of kafka. >> >> That seems like the ideal place to do this. >> >> >> >> >> >> > On 03 Jun 2016, at 07:27, Jim Hoagland <jim_hoagl...@symantec.com >> <javascript:;>> >> >>wrote: >> >> > >> >> > I'm hesitant to cite it because it wasn't really a proper >>benchmark, >> >>but >> >> > with the end-to-end encryption through Kafka proof of concept >> >>described at >> >> > http://symc.ly/1pC2CEG, doing the encryption added only 26% to the >> >>time >> >> > taken to send messages and only 6% to the time taken to consume >> >>messages. >> >> > This is with batching 200 300-byte messages together for >>encryption. >> >>More >> >> > details are in the blog post. >> >> > >> >> > Personally I think that encrypting sensitive data before handing >>it to >> >> > Kafka (or at least before it leaves the producing box) just makes >> >>sense to >> >> > do if the situation allows. The Kafka installation wouldn't be >>able >> >> > reveal the data even if its systems and networks are compromised >> >>because >> >> > it never sees the data in the clear and doesn't know how to decrypt >> >>it. >> >> > In the way we set it up, someone would need the recipient's RSA >> >>private >> >> > key to decrypt (or would need to have compromised a decrypting >> >>system). >> >MG>Jim can we assume you only implement Asymmetric Cryptography? >> >> > >> >> > -- Jim >> >> > >> >> > >> >> > On 6/2/16, 2:56 AM, "Tom Crayford" <tcrayf...@heroku.com >> <javascript:;>> wrote: >> >> > >> >> >> Filesystem encryption is transparent to Kafka. You don't need to >>use >> >>SSL, >> >> >> but your encryption requirements may cause you to need SSL as >>well. >> >> >> >> >> >> With regards to compression, without adding at rest encryption to >> >>Kafka >> >> >> (which is a very major piece of work, one that for sure requires a >> >>KIP and >> >> >> has many, many implications), there's not much to do there. I >>think >> >>it's >> >> >> worth examining your threat models that require encryption on disk >> >>without >> >> >> full disk encryption being suitable. Generally compromised broker >> >>machines >> >> >> means an attacker will be able to sniff in flight traffic anyway, >>if >> >>the >> >> >> goal is to never leak messages even if an attacker has full >>control >> >>of the >> >> >> broker machine, I'd suggest that that seems pretty impossible >>under >> >> >> current >> >> >> operating environments. >> >> >> >> >> >> If the issue is compliance, I'd recommend querying whichever >> >>compliance >> >> >> standard you're operating under about the suitability of full disk >> >> >> encryption, and careful thought about encrypting the most >>sensitive >> >>parts >> >> >> of messages. Whilst encryption in the producer and consumer does >> >>lead to >> >> >> performance issues and decrease the capability of compression to >> >>shrink a >> >> >> dataset, doing partial encryption of messages is easy enough. >> >> >> >> >> >> Generally we've found that the kinds of uses of Kafka that >>require in >> >> >> message encryption (alongside full disk encryption and SSL which >>we >> >> >> provide >> >> >> as standard) don't have such high throughput needs that they worry >> >>about >> >> >> compression etc. That clearly isn't true for all use cases though. >> >> >> >> >> >> Thanks >> >> >> >> >> >> Tom Crayford >> >> >> Heroku Kafka >> >> >> >> >> >> On Thursday, 2 June 2016, Gerard Klijs <gerard.kl...@dizzit.com >> <javascript:;>> >> >>wrote: >> >> >> >> >> >>> You could add a header to every message, with information whether >> >>it's >> >> >>> encrypted or not, then you don't have to encrypt all the >>messages, >> >>or >> >> >>> you >> >> >>> only do it for some topics. >> >> >>> >> >> >>> On Thu, Jun 2, 2016 at 6:36 AM Bruno Rassaerts < >> >> >>> bruno.rassae...@novazone.be <javascript:;> <javascript:;>> >> >> >>> wrote: >> >> >>> >> >> >>>> It works indeed but encrypting individual messages really >> >>influences >> >> >>> the >> >> >>>> batch compression done by Kafka. >> >> >>>> Performance drops to about 1/3 of what it is without (even if we >> >> >>> prepare >> >> >>>> the encrypted samples upfront). >> >> >>>> In the end what we going for is only encrypting what we really >> >>really >> >> >>> need >> >> >>>> to encrypt, not every message systematically. >> >> >>>> >> >> >>>>> On 31 May 2016, at 13:00, Gerard Klijs <gerard.kl...@dizzit.com >> <javascript:;> >> >> >>> <javascript:;>> wrote: >> >> >>>>> >> >> >>>>> If you want system administrators not being able to see the >>data, >> >> >>> the >> >> >>>> only >> >> >>>>> option is encryption, with only the clients sharing the key (or >> >> >>> whatever >> >> >>>> is >> >> >>>>> used to (de)crypt the data). Like the example from eugene. I >>don't >> >> >>> know >> >> >>>> the >> >> >>>>> kind of messages you have, but you could always wrap something >> >> >>> around >> >> >>> any >> >> >>>>> (de)serializer your currently using. >> >> >>>>> >> >> >>>>> On Tue, May 31, 2016 at 12:21 PM Bruno Rassaerts < >> >> >>>>> bruno.rassae...@novazone.be <javascript:;> <javascript:;>> >>wrote: >> >> >>>>> >> >> >>>>>> I’ve asked the same question in the past, and disk encryption >>was >> >> >>>>>> suggested as a solution as well. >> >> >>>>>> However, as far as I know, disk encryption will not prevent >>your >> >> >>> data >> >> >>> to >> >> >>>>>> be stolen when the machine is compromised. >> >> >>>>>> What we are looking for is even an additional barrier, so that >> >>even >> >> >>>> system >> >> >>>>>> administrators do not have access to the data. >> >> >>>>>> Any suggestions ? >> >> >>>>>> >> >> >>>>>>> On 24 May 2016, at 14:40, Tom Crayford <tcrayf...@heroku.com >> <javascript:;> >> >> >>> <javascript:;>> wrote: >> >> >>>>>>> >> >> >>>>>>> Hi, >> >> >>>>>>> >> >> >>>>>>> There's no encryption at rest. It's recommended to use >> >>filesystem >> >> >>>>>>> encryption, or encryption of each individual message before >> >> >>> producing >> >> >>>> it >> >> >>>>>>> for this. >> >> >>>>>>> >> >> >>>>>>> Only the new producer and consumers have SSL support. >> >> >>>>>>> >> >> >>>>>>> Thanks >> >> >>>>>>> >> >> >>>>>>> Tom Crayford >> >> >>>>>>> Heroku Kafka >> >> >>>>>>> >> >> >>>>>>> On Tue, May 24, 2016 at 11:33 AM, Snehalata Nagaje < >> >> >>>>>>> snehalata.nag...@harbingergroup.com <javascript:;> >> <javascript:;>> wrote: >> >> >>>>>>> >> >> >>>>>>>> >> >> >>>>>>>> >> >> >>>>>>>> Thanks for quick reply. >> >> >>>>>>>> >> >> >>>>>>>> Do you mean If I see messages in kafka, those will not be >> >> >>> readable? >> >> >>>>>>>> >> >> >>>>>>>> And also, we are using new producer but old consumer , does >>old >> >> >>>> consumer >> >> >>>>>>>> have ssl support? >> >> >>>>>>>> >> >> >>>>>>>> As mentioned in document, its not there. >> >> >>>>>>>> >> >> >>>>>>>> >> >> >>>>>>>> Thanks, >> >> >>>>>>>> Snehalata >> >> >>>>>>>> >> >> >>>>>>>> ----- Original Message ----- >> >> >>>>>>>> From: "Mudit Kumar" <mudit.ku...@askme.in <javascript:;> >> <javascript:;>> >> >> >>>>>>>> To: users@kafka.apache.org <javascript:;> <javascript:;> >> >> >>>>>>>> Sent: Tuesday, May 24, 2016 3:53:26 PM >> >> >>>>>>>> Subject: Re: Kafka encryption >> >> >>>>>>>> >> >> >>>>>>>> Yes,it does that.What specifically you are looking for? >> >> >>>>>>>> >> >> >>>>>>>> >> >> >>>>>>>> >> >> >>>>>>>> >> >> >>>>>>>> On 5/24/16, 3:52 PM, "Snehalata Nagaje" < >> >> >>>>>>>> snehalata.nag...@harbingergroup.com <javascript:;> >> <javascript:;>> wrote: >> >> >>>>>>>> >> >> >>>>>>>>> Hi All, >> >> >>>>>>>>> >> >> >>>>>>>>> >> >> >>>>>>>>> We have requirement of encryption in kafka. >> >> >>>>>>>>> >> >> >>>>>>>>> As per docs, we can configure kafka with ssl, for secured >> >> >>>>>> communication. >> >> >>>>>>>>> >> >> >>>>>>>>> But does kafka also stores data in encrypted format? >> >> >>>>>>>>> >> >> >>>>>>>>> >> >> >>>>>>>>> Thanks, >> >> >>>>>>>>> Snehalata >> >> >>>>>>>> >> >> >>>>>> >> >> >>>>>> >> >> >>>> >> >> >>>> >> >> >>> >> >> > >> >> >> > >> >>
