Hi Sönke, Replies inline
1. The functionality in this first phase could indeed be achieved with > custom serializers, that would then need to wrap the actual serializer that > is to be used. However, looking forward I intend to add functionality that > allows configuration to be configured broker-side via topic level configs > and investigate encrypting entire batches of messages for performance. Both > those things would require us to move past doing this in a serializer, so I > think we should take that plunge now to avoid unnecessary refactoring later > on. > I suspect you might have a hard time getting this KIP approved when the immediate use cases it serves can already be implemented using custom serialization. Having a working implementation using custom serialization would: * prove there's interest in these features amongst end users * prove that there's interest in the specific features which would require end-to-end encryption to be implemented in Kafka itself * validate that the interfaces/abstractions in this proposal are the right ones All of those things would strengthen the argument for getting this into Apache Kafka eventually. > 2. Absolutely! I am currently working on a very (very) rough implementation > to kind of prove the principle. I'll add those to the KIP as soon as I > think they are in a somewhat final form. > There are a lot of design details missing from the KIP, I didn't want to go > all the way just for people to hate what I designed and have to start over > ;) > > 3. Yes. I plan to create a LocalKeystoreKeyManager (name tbd) as part of > this KIP that allows configuring keys per topic pattern and will read the > keys from a local file. This will provide encryption, but users would have > to manually sync keystores across consumer and producer systems. Proper key > management with rollover and retrieval from central vaults would come in a > later phase. > I think this is the hard part in many respects. Having a working implementation for at least one key management system would presumably be a prerequisite for getting this merged. Even if this KIP got merged I think it's likely that there would be a desire to limit the number of implementations of the interfaces within Apache Kafka because of the maintenance and testing burden. (We've seen this in other areas previously, ConfigProviders being one example.) So again, this suggests to me that you might make more progress implementing this outside Apache Kafka for the moment. Having said all that, these are just my thoughts second guessing what the community might do. I might be wrong. Kind regards, Tom
