We are trying to deploy kafka into EC2 and one of the requirement from infosec is to have kafka encryption at rest (stored with encrypted value). We also need to be able to rotate encryption keys and re-encrypt all the messages on regular basis since we are a financial company. The re-encryption feels challenging since kafka messages are immutable from client side (producer and consumer). Some ideas floating around to have replicated clustered but then it will mess up all the offsets of the consumer and switching is complicated from operational perspective.
One idea we have is to achieve this is to plugin our own "compression" codec which deal with both compression and encryption logic and leverage compaction cycle to re-write all the messages by calling decompress and compress into a new file. It feels this approach can also have zero impact to the consumer/producer if they are using the same "codec" for compression since the offsets will be intact. My current understanding is the codecs are hardcoded right now (we are using .9) so it will require us to customize kafka. Also compaction cannot be triggered on-demand, which is needed in case of the key loss. So before we take on customization of kafka, I am just wondering if our thinking is on the right track. I hope some of the committers from Confluent/Hornton/Cloudera can comment on that and the road map to support encryption at rest and key rotation, or otherwise alternative to what is proposed. Also please let me know if my question/problem is not clear. Thanks, Josh ________________________________ DISCLAIMER: The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. Any review, re-transmission, dissemination or other use of, or taking of any action in reliance upon this information by persons or entities other than the intended recipient is prohibited. If you received this in error, please contact the sender and destroy any copies of this document and any attachments.
