Hello, In our project we have a very strong requirement to protect all data, all the time. Even when the data is “in-rest” on disk, it needs to be protected. We’ve been trying to figure out how to this with Kafka, and hit some obstacles.
One thing we’ve tried to do is to encrypt every message we hand over to kafka. This results in the encrypted messages being written to disk on the brokers. However, the performance of performing encryption has serious performance implications, due to the CPU intensive operation which encryption is, and the fact that batch compression offered by Kafka is not nearly as efficient anymore after encrypting the data. Doing this message by message encryption gives us a performance penalty of about 75%, even if we compress the messages before encryption. What we are looking for is a way to plugin our encryption in two possible locations: 1. As a custom compression algorithm, which would batch compress, and batch encrypt. And get the files stored as such. 2. As a encryption plugin specifically designed for storing the kafka broker files. Is there any way that this can be done using Kafka (0.9), or can somebody point us to the place were we could add this in the Kafka codebase. Thanks, Bruno Rassaerts
