Hi again, Of course I was overlooking at least one thing. Anyone who could guess the record keys could hash them and compare. To make it work the producer and consumer would need a shared secret to include in the hash computation. But the key management service could furnish them with this in addition to the encryption/decryption keys, so I think it should still work. Unless I've overlooked something else.
Cheers, Tom On Thu, May 7, 2020 at 6:04 PM Tom Bentley <tbent...@redhat.com> wrote: > Hi Rayanne, > > You raise some good points there. > > Similarly, if the whole record is encrypted, it becomes impossible to do >> joins, group bys etc, which just need the record key and maybe don't have >> access to the encryption key. Maybe only record _values_ should be >> encrypted, and maybe Kafka Streams could defer decryption until the actual >> value is inspected. That way joins etc are possible without the encryption >> key, and RocksDB would not need to decrypt values before materializing to >> disk. >> > > It's getting a bit late here, so maybe I overlooked something, but > wouldn't the natural thing to do be to make the "encrypted" key a hash of > the original key, and let the value of the encrypted value be the cipher > text of the (original key, original value) pair. A scheme like this would > preserve equality of the key (strictly speaking there's a chance of > collision of course). I guess this could also be a solution for the > compacted topic issue Sönke mentioned. > > Cheers, > > Tom > > > > On Thu, May 7, 2020 at 5:17 PM Ryanne Dolan <ryannedo...@gmail.com> wrote: > >> Thanks Sönke, this is an area in which Kafka is really, really far behind. >> >> I've built secure systems around Kafka as laid out in the KIP. One issue >> that is not addressed in the KIP is re-encryption of records after a key >> rotation. When a key is compromised, it's important that any data >> encrypted >> using that key is immediately destroyed or re-encrypted with a new key. >> Ideally first-class support for end-to-end encryption in Kafka would make >> this possible natively, or else I'm not sure what the point would be. It >> seems to me that the brokers would need to be involved in this process, so >> perhaps a client-first approach will be painting ourselves into a corner. >> Not sure. >> >> Another issue is whether materialized tables, e.g. in Kafka Streams, would >> see unencrypted or encrypted records. If we implemented the KIP as >> written, >> it would still result in a bunch of plain text data in RocksDB everywhere. >> Again, I'm not sure what the point would be. Perhaps using custom serdes >> would actually be a more holistic approach, since Kafka Streams etc could >> leverage these as well. >> >> Similarly, if the whole record is encrypted, it becomes impossible to do >> joins, group bys etc, which just need the record key and maybe don't have >> access to the encryption key. Maybe only record _values_ should be >> encrypted, and maybe Kafka Streams could defer decryption until the actual >> value is inspected. That way joins etc are possible without the encryption >> key, and RocksDB would not need to decrypt values before materializing to >> disk. >> >> This is why I've implemented encryption on a per-field basis, not at the >> record level, when addressing kafka security in the past. And I've had to >> build external pipelines that purge, re-encrypt, and re-ingest records >> when >> keys are compromised. >> >> This KIP might be a step in the right direction, not sure. But I'm >> hesitant >> to support the idea of end-to-end encryption without a plan to address the >> myriad other problems. >> >> That said, we need this badly and I hope something shakes out. >> >> Ryanne >> >> On Tue, Apr 28, 2020, 6:26 PM Sönke Liebau >> <soenke.lie...@opencore.com.invalid> wrote: >> >> > All, >> > >> > I've asked for comments on this KIP in the past, but since I didn't >> really >> > get any feedback I've decided to reduce the initial scope of the KIP a >> bit >> > and try again. >> > >> > I have reworked to KIP to provide a limited, but useful set of features >> for >> > this initial KIP and laid out a very rough roadmap of what I'd envision >> > this looking like in a final version. >> > >> > I am aware that the KIP is currently light on implementation details, >> but >> > would like to get some feedback on the general approach before fully >> > speccing everything. >> > >> > The KIP can be found at >> > >> > >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-317%3A+Add+end-to-end+data+encryption+functionality+to+Apache+Kafka >> > >> > >> > I would very much appreciate any feedback! >> > >> > Best regards, >> > Sönke >> > >> >
