"We need to be capable of changing encryption keys on regular intervals and in case of expected key compromise." is achievable with full disk encryption particularly if you are willing to add and remove Kafka servers so that you replicate the data to new machines/disks with new keys and take the machines with old keys out of use and wipe them.
For the second part of it I would suggest reevaluating your threat model since you are looking at a machine that is compromised but not compromised enough to be able to read the key from Kafka or to use Kafka to read the data. While you could add support to encrypt data on the way in and out of compression I believe you would need either substantial work in Kafka to support rewriting/reencrypting the logfiles (with performance penalties) or rotate machines in and out as with full disk encryption. Though I'll let someone with more knowledge of the implementation comment further on what would be required. Christian On Mon, May 2, 2016 at 9:41 PM, Bruno Rassaerts <bruno.rassae...@novazone.be> wrote: > We did try indeed the last scenario you describe as encrypted disks do not > fulfil our requirements. > We need to be capable of changing encryption keys on regular intervals and in > case of expected key compromise. > Also, when a running machine is hacked, disk based or file system based > encryption doesn’t offer any protection. > > Our goal is indeed to have the content in the broker files encrypted. The > problem is that only way to achieve this is through custom serialisers. > This works, but the overhead is quite dramatic as the messages are no longer > efficiently compressed (in batch). > Compression in the serialiser, before the encryption, doesn’t really solve > the performance problem. > > The best thing for us would be able to encrypt after the batch compression > offered by kafka. > The hook to do this is missing in the current implementation. > > Bruno > >> On 02 May 2016, at 22:46, Tom Brown <tombrow...@gmail.com> wrote: >> >> I'm trying to understand your use-case for encrypted data. >> >> Does it need to be encrypted only over the wire? This can be accomplished >> using TLS encryption (v0.9.0.0+). See >> https://issues.apache.org/jira/browse/KAFKA-1690 >> >> Does it need to be encrypted only when at rest? This can be accomplished >> using full disk encryption as others have mentioned. >> >> Does it need to be encrypted during both? Use both TLS and full disk >> encryption. >> >> Does it need to be encrypted fully from end-to-end so even Kafka can't read >> it? Since Kafka shouldn't be able to know the contents, the key should not >> be known to Kafka. What remains is manually encrypting each message before >> giving it to the producer (or by implementing an encrypting serializer). >> Either way, each message is still encrypted individually. >> >> Have I left out a scenario? >> >> --Tom >> >> >> On Mon, May 2, 2016 at 2:01 PM, Bruno Rassaerts <bruno.rassae...@novazone.be >>> wrote: >> >>> Hello, >>> >>> We tried encrypting the data before sending it to kafka, however this >>> makes the compression done by kafka almost impossible. >>> Also the performance overhead of encrypting the individual messages was >>> quite significant. >>> >>> Ideally, a pluggable “compression” algorithm would be best. Where message >>> can first be compressed, then encrypted in batch. >>> However, the current kafka implementation does not allow this. >>> >>> Bruno >>> >>>> On 26 Apr 2016, at 19:02, Jim Hoagland <jim_hoagl...@symantec.com> >>> wrote: >>>> >>>> Another option is to encrypt the data before you hand it to Kafka and >>> have >>>> the downstream decrypt it. This takes care of on-disk on on-wire >>>> encryption. We did a proof of concept of this: >>>> >>>> >>> http://www.symantec.com/connect/blogs/end-end-encryption-though-kafka-our-p >>>> roof-concept >>>> >>>> ( http://symc.ly/1pC2CEG ) >>>> >>>> -- Jim >>>> >>>> On 4/25/16, 11:39 AM, "David Buschman" <david.busch...@timeli.io> wrote: >>>> >>>>> Kafka handles messages which are compose of an array of bytes. Kafka >>> does >>>>> not care what is in those byte arrays. >>>>> >>>>> You could use a custom Serializer and Deserializer to encrypt and >>> decrypt >>>>> the data from with your application(s) easily enough. >>>>> >>>>> This give the benefit of having encryption at rest and over the wire. >>> Two >>>>> birds, one stone. >>>>> >>>>> DaVe. >>>>> >>>>> >>>>>> On Apr 25, 2016, at 2:14 AM, Jens Rantil <jens.ran...@tink.se> wrote: >>>>>> >>>>>> IMHO, I think that responsibility should lie on the file system, not >>>>>> Kafka. >>>>>> Feels like a waste of time and double work to implement that unless >>>>>> there's >>>>>> a really good reason for it. Let's try to keep Kafka a focused product >>>>>> that >>>>>> does one thing well. >>>>>> >>>>>> Cheers, >>>>>> Jens >>>>>> >>>>>> On Fri, Apr 22, 2016 at 3:31 AM Tauzell, Dave >>>>>> <dave.tauz...@surescripts.com> >>>>>> wrote: >>>>>> >>>>>>> I meant encryption of the data at rest. We utilize filesytem >>>>>>> encryption >>>>>>> for other products; just wondering if anything was on the Kafka >>>>>>> roadmap. >>>>>>> >>>>>>> Dave >>>>>>> >>>>>>>> On Apr 21, 2016, at 18:12, Martin Gainty <mgai...@hotmail.com> >>> wrote: >>>>>>>> >>>>>>>> Dave- >>>>>>>> so you want username/password credentials to be sent in response to >>> an >>>>>>> HTTP Get as clear text? >>>>>>>> if not this has been asked and answered with Axishttps:// >>>>>>> axis.apache.org/axis2/java/rampart/ >>>>>>>> >>>>>>>> Martin >>>>>>>> ______________________________________________ >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> From: dave.tauz...@surescripts.com >>>>>>>>> To: users@kafka.apache.org >>>>>>>>> Subject: Encryption at Rest >>>>>>>>> Date: Thu, 21 Apr 2016 21:31:56 +0000 >>>>>>>>> >>>>>>>>> Has there been any discussion or work on at rest encryption for >>>>>>>>> Kafka? >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> Dave >>>>>>>>> >>>>>>>>> This e-mail and any files transmitted with it are confidential, may >>>>>>> contain sensitive information, and are intended solely for the use of >>>>>>> the >>>>>>> individual or entity to whom they are addressed. If you have received >>>>>>> this >>>>>>> e-mail in error, please notify the sender by reply e-mail immediately >>>>>>> and >>>>>>> destroy all copies of the e-mail and any attachments. >>>>>>>> >>>>>>> >>>>>> -- >>>>>> >>>>>> Jens Rantil >>>>>> Backend Developer @ Tink >>>>>> >>>>>> Tink AB, Wallingatan 5, 111 60 Stockholm, Sweden >>>>>> For urgent matters you can reach me at +46-708-84 18 32. >>>>> >>>> >>> >>> >
