Thanks for your feedback both of you! I've commented inline below.
On Thu, 8 Aug 2019 at 08:38, Jörn Franke <jornfra...@gmail.com> wrote: > If you are doing batch encryption then you are more similar to a scenario > of file encryption. The more frequent the messages are you are closer to > the ssl/https scenarios. You may learn from those protocols on how they > handle keys, how long they keep them etc. to implement your E2e solution . > > > Am 08.08.2019 um 08:11 schrieb Maulin Vasavada < > maulin.vasav...@gmail.com>: > > > > Hi Sönke Liebau > > < > https://www.mail-archive.com/search?l=dev@kafka.apache.org&q=from:%22S%C3%B6nke+Liebau%22 > > > > > > Thanks for the great detailed documentation. However, I feel by leaving > the > > KMS outside of Kafka might simplify the whole thing to a great extent. If > > the broker is not going to touch the encrypted messages, why would we put > > any dependency of KMS interfaces on the Broker. We have experimented > doing > > end-to-end message encryption and we used topic level keys and message > > encryption with serializer wrapper which encrypts each message before > > serializing. The serializer wrapper have to integrate with required KMS > we > > use internally and that was all. > My idea by having the broker manage topic keys was that we keep the option of actually making the encryption transparent to the clients. This way you could configure a topic as encrypted on the broker and the broker would then push everything to the client that it needs to know to encrypt messages on startup - but still be unable to decrypt messages itself. However, this is only one possible scenario. Another valid scenario is of course that you want to configure clients directly with keys, which I hope my proposal also covers, as everything is pluggable. And in this case the broker would not need a dependency on the KMS, as it doesn't need to handle keys. Basically by making this pluggable I hope to be able to cover a wide variety of use cases, the two described in the KIP are just the ones that I'd implement initially. > > > > However one key observation we had was - if we could do encryption at > > 'batch' level instead of 'per-message' it can perform much better > > (depending upon batch sizing). We didn't experiment with that though. > I agree, batch encryption would make this perform much better, but it has downsides as well. I am unsure of the security implications of larger vs smaller payload to be honest, but will investigate this. In addition however, we do not want to decrypt the batch on the broker, so this will be handed to consumers as a batch as well, which has the same implications as end-to-end compression like more complicated offset committing for consumers. I have not looked into that in a long time and that may not even be an issue anymore. I'll do some digging here as well. Bottom line: I agree, but I think we should offer both modes of operation. > > > > Thanks > > Maulin > -- Sönke Liebau Partner Tel. +49 179 7940878 OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880 Wedel - Germany
