I'm trying to understand your use-case for encrypted data. Does it need to be encrypted only over the wire? This can be accomplished using TLS encryption (v0.9.0.0+). See https://issues.apache.org/jira/browse/KAFKA-1690
Does it need to be encrypted only when at rest? This can be accomplished using full disk encryption as others have mentioned. Does it need to be encrypted during both? Use both TLS and full disk encryption. Does it need to be encrypted fully from end-to-end so even Kafka can't read it? Since Kafka shouldn't be able to know the contents, the key should not be known to Kafka. What remains is manually encrypting each message before giving it to the producer (or by implementing an encrypting serializer). Either way, each message is still encrypted individually. Have I left out a scenario? --Tom On Mon, May 2, 2016 at 2:01 PM, Bruno Rassaerts <bruno.rassae...@novazone.be > wrote: > Hello, > > We tried encrypting the data before sending it to kafka, however this > makes the compression done by kafka almost impossible. > Also the performance overhead of encrypting the individual messages was > quite significant. > > Ideally, a pluggable “compression” algorithm would be best. Where message > can first be compressed, then encrypted in batch. > However, the current kafka implementation does not allow this. > > Bruno > > > On 26 Apr 2016, at 19:02, Jim Hoagland <jim_hoagl...@symantec.com> > wrote: > > > > Another option is to encrypt the data before you hand it to Kafka and > have > > the downstream decrypt it. This takes care of on-disk on on-wire > > encryption. We did a proof of concept of this: > > > > > http://www.symantec.com/connect/blogs/end-end-encryption-though-kafka-our-p > > roof-concept > > > > ( http://symc.ly/1pC2CEG ) > > > > -- Jim > > > > On 4/25/16, 11:39 AM, "David Buschman" <david.busch...@timeli.io> wrote: > > > >> Kafka handles messages which are compose of an array of bytes. Kafka > does > >> not care what is in those byte arrays. > >> > >> You could use a custom Serializer and Deserializer to encrypt and > decrypt > >> the data from with your application(s) easily enough. > >> > >> This give the benefit of having encryption at rest and over the wire. > Two > >> birds, one stone. > >> > >> DaVe. > >> > >> > >>> On Apr 25, 2016, at 2:14 AM, Jens Rantil <jens.ran...@tink.se> wrote: > >>> > >>> IMHO, I think that responsibility should lie on the file system, not > >>> Kafka. > >>> Feels like a waste of time and double work to implement that unless > >>> there's > >>> a really good reason for it. Let's try to keep Kafka a focused product > >>> that > >>> does one thing well. > >>> > >>> Cheers, > >>> Jens > >>> > >>> On Fri, Apr 22, 2016 at 3:31 AM Tauzell, Dave > >>> <dave.tauz...@surescripts.com> > >>> wrote: > >>> > >>>> I meant encryption of the data at rest. We utilize filesytem > >>>> encryption > >>>> for other products; just wondering if anything was on the Kafka > >>>> roadmap. > >>>> > >>>> Dave > >>>> > >>>>> On Apr 21, 2016, at 18:12, Martin Gainty <mgai...@hotmail.com> > wrote: > >>>>> > >>>>> Dave- > >>>>> so you want username/password credentials to be sent in response to > an > >>>> HTTP Get as clear text? > >>>>> if not this has been asked and answered with Axishttps:// > >>>> axis.apache.org/axis2/java/rampart/ > >>>>> > >>>>> Martin > >>>>> ______________________________________________ > >>>>> > >>>>> > >>>>> > >>>>>> From: dave.tauz...@surescripts.com > >>>>>> To: users@kafka.apache.org > >>>>>> Subject: Encryption at Rest > >>>>>> Date: Thu, 21 Apr 2016 21:31:56 +0000 > >>>>>> > >>>>>> Has there been any discussion or work on at rest encryption for > >>>>>> Kafka? > >>>>>> > >>>>>> Thanks, > >>>>>> Dave > >>>>>> > >>>>>> This e-mail and any files transmitted with it are confidential, may > >>>> contain sensitive information, and are intended solely for the use of > >>>> the > >>>> individual or entity to whom they are addressed. If you have received > >>>> this > >>>> e-mail in error, please notify the sender by reply e-mail immediately > >>>> and > >>>> destroy all copies of the e-mail and any attachments. > >>>>> > >>>> > >>> -- > >>> > >>> Jens Rantil > >>> Backend Developer @ Tink > >>> > >>> Tink AB, Wallingatan 5, 111 60 Stockholm, Sweden > >>> For urgent matters you can reach me at +46-708-84 18 32. > >> > > > >
