Thanks Sönke, this is an area in which Kafka is really, really far behind.
I've built secure systems around Kafka as laid out in the KIP. One issue that is not addressed in the KIP is re-encryption of records after a key rotation. When a key is compromised, it's important that any data encrypted using that key is immediately destroyed or re-encrypted with a new key. Ideally first-class support for end-to-end encryption in Kafka would make this possible natively, or else I'm not sure what the point would be. It seems to me that the brokers would need to be involved in this process, so perhaps a client-first approach will be painting ourselves into a corner. Not sure. Another issue is whether materialized tables, e.g. in Kafka Streams, would see unencrypted or encrypted records. If we implemented the KIP as written, it would still result in a bunch of plain text data in RocksDB everywhere. Again, I'm not sure what the point would be. Perhaps using custom serdes would actually be a more holistic approach, since Kafka Streams etc could leverage these as well. Similarly, if the whole record is encrypted, it becomes impossible to do joins, group bys etc, which just need the record key and maybe don't have access to the encryption key. Maybe only record _values_ should be encrypted, and maybe Kafka Streams could defer decryption until the actual value is inspected. That way joins etc are possible without the encryption key, and RocksDB would not need to decrypt values before materializing to disk. This is why I've implemented encryption on a per-field basis, not at the record level, when addressing kafka security in the past. And I've had to build external pipelines that purge, re-encrypt, and re-ingest records when keys are compromised. This KIP might be a step in the right direction, not sure. But I'm hesitant to support the idea of end-to-end encryption without a plan to address the myriad other problems. That said, we need this badly and I hope something shakes out. Ryanne On Tue, Apr 28, 2020, 6:26 PM Sönke Liebau <soenke.lie...@opencore.com.invalid> wrote: > All, > > I've asked for comments on this KIP in the past, but since I didn't really > get any feedback I've decided to reduce the initial scope of the KIP a bit > and try again. > > I have reworked to KIP to provide a limited, but useful set of features for > this initial KIP and laid out a very rough roadmap of what I'd envision > this looking like in a final version. > > I am aware that the KIP is currently light on implementation details, but > would like to get some feedback on the general approach before fully > speccing everything. > > The KIP can be found at > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-317%3A+Add+end-to-end+data+encryption+functionality+to+Apache+Kafka > > > I would very much appreciate any feedback! > > Best regards, > Sönke >
