You could do this with (I expect) reasonable efficiency and with no changes to Kafka code by using multiple topics.
You can have a script that in a streaming manner reads out all messages in a topic, decrypts them with the old key, encrypts them with the new key, and adds them to a new topic. At the time you need to re-encrypt all the messages in a topic, invoke the script. You would probably have one topic per encryption key. The producers would know the right topic to write to since they know which version of the encryption key. The consumers would have to have some logic to switch over. Perhaps the last message in a particular topic could tell the consumers to switch to a different topic. If you invoke the script from someplace close to the Kafka brokers, the overhead should I think not be too much higher than custom codec approach you were talking about. Just some local network traffic, with the amount depending on the size of the topic and your replication factor. -- Jim On 1/20/16, 10:37 AM, "Josh Wo" <z...@lendingclub.com> wrote: >Hi Jens, >I got your point but some of our use case cannot just rely on TTL. We try >to have long expiry for message and rather compact them (dedup) so we can >replay messages as system of records. When key is lost, we will invalid >the old key so message encrypted by old message will not be able to >decrypt if we don't re-encrypt them also. > >Josh >________________________________________ >From: Jens Rantil <jens.ran...@tink.se> >Sent: Tuesday, January 19, 2016 11:48 PM >To: users@kafka.apache.org >Cc: users@kafka.apache.org >Subject: Re: security: encryption at rest and key rotation idea > >Hi Josh, > > >Kafka will/can expire message logs after a certain TTL. You can't simply >rely on expiration for key rotation? That is, you start to produce >messages with a different key while your consumer temporarily handles the >overlap of keys for the duration of the TTL. > > > > >Just an idea, > >Jens > > > > > >– >Skickat från Mailbox > >On Wed, Jan 20, 2016 at 12:34 AM, Josh Wo <z...@lendingclub.com> wrote: > >> We are trying to deploy kafka into EC2 and one of the requirement from >>infosec is to have kafka encryption at rest (stored with encrypted >>value). We also need to be able to rotate encryption keys and re-encrypt >>all the messages on regular basis since we are a financial company. The >>re-encryption feels challenging since kafka messages are immutable from >>client side (producer and consumer). Some ideas floating around to have >>replicated clustered but then it will mess up all the offsets of the >>consumer and switching is complicated from operational perspective. >> One idea we have is to achieve this is to plugin our own "compression" >>codec which deal with both compression and encryption logic and leverage >>compaction cycle to re-write all the messages by calling decompress and >>compress into a new file. It feels this approach can also have zero >>impact to the consumer/producer if they are using the same "codec" for >>compression since the offsets will be intact. >> My current understanding is the codecs are hardcoded right now (we are >>using .9) so it will require us to customize kafka. Also compaction >>cannot be triggered on-demand, which is needed in case of the key loss. >>So before we take on customization of kafka, I am just wondering if our >>thinking is on the right track. >> I hope some of the committers from Confluent/Hornton/Cloudera can >>comment on that and the road map to support encryption at rest and key >>rotation, or otherwise alternative to what is proposed. Also please let >>me know if my question/problem is not clear. >> Thanks, >> Josh >> ________________________________ >> DISCLAIMER: The information transmitted is intended only for the person >>or entity to which it is addressed and may contain confidential and/or >>privileged material. Any review, re-transmission, dissemination or other >>use of, or taking of any action in reliance upon this information by >>persons or entities other than the intended recipient is prohibited. If >>you received this in error, please contact the sender and destroy any >>copies of this document and any attachments.
