Hi all, I have created KIP-317 [1] a while ago, which outlines an implementation proposal to add transparent data encryption functionality to Kafka. The KIP in its current form is somewhat rigid in its implementation, I will rework this to become extensible over the next few days to allow for additional implementations.
I have discussed the current method of providing keys with a colleague and while we agree that this is a valid use case for some people, there are certainly a lot of other valid use cases out there as well. To ensure that the initial implementation provides the necessary flexibility I'd like some feedback from the community on what requirements they would have around data encryption and key management. The following questions should serve as a starting point for the discussion, please feel free to address anything that comes to mind which I have not mentioned here: - Should encryption be configurable rather on the client or on the broker and be pushed down to the client? - Where should keys be stored? - How much flexibility around keys is necessary - is there for example a use case that would decide on a per message basis which key to use? (imagine a topic containing top secret, secret and public data with three different keys) - Do we need functionality to prohibit publishing unencrypted messages to topics based on that topics setup? Of course the mailing list is the first place that discussions like these should take place, but sometimes I find a face to face discussion can be quite useful as well, especially when discussing non-trivial topics (like encryption). I have reached out to the organizers of the upcoming Kafka Summit in SF and there might be a chance for us to get a room with a whiteboard at some point (probably during lunch, when the room is otherwise unused). Would people be interested in meeting up for 20 minutes to discuss this in person? I'd be happy to provide a summary on the mailing list afterwards of course. Look forward to hearing from all of you! Best regards, Sönke [1] https://cwiki.apache.org/confluence/display/KAFKA/KIP-317%3A+Add+transparent+data+encryption+functionality
