Proposal for an End-to-End encryption in Kafka: https://github.com/salyh/kafka/commit/151a0340645f4165cb120f79df544326e5a0db6f
Introduce EncryptingSerializer and DecryptingDeserializer. They act as wrappers for inner (de)serializer and provide transparent AES and RSA based end-to-end encryption. So all messages stored in Kafka which passed the EncryptingSerializer are stored in an encrypted manner. The DecryptingDeserializer can detect if a message is encrypted and if not its just passed through as it is. Here is how it works: https://github.com/salyh/kafka/blob/151a0340645f4165cb120f79df544326e5a0db6f/clients/src/main/java/org/apache/kafka/common/serialization/EncrpytingSerializer.java Currently the code is not threadsafe and not optimized or benchmarked. Its just a proof of concept to start a discussion if such a contribution would be useful. Depending on the RSA keysize a overhead of additional 41+(RSA keysize in bytes) bytes per message are added and for each message produced (or consumed) a AES en-/decryption needs to be performed. RSA encryption of the AES key happens not very often cause this could be cached - so RSA en-/decryption is likely not a big issue regarding performance. Note: AES encryption on modern hardware with a recent Java runtime is typically hardware accelerated (by leveraging AES-NI and -XX:+UseAES -XX:+UseAESIntrinsics) Feedback and comments appreciated. Thanks Hendrik -- Hendrik Saly (salyh, hendrikdev22) @hendrikdev22 PGP: 0x22D7F6EC salyh <AT> apache <DOT> org
