Re: Key rotation in Iceberg data encryption

2021-03-25 Thread Gidon Gershinsky
Sounds good. Giving the users a tool, and the decision to make on whether to rotate a KEK and replace the manifest file, is a flexible way to address this for now. As we gather more information on the safety of unrotated KEKs, and on the consequences of replacing the manifest files, we can either d

Re: Key rotation in Iceberg data encryption

2021-03-25 Thread Ye, Jack
Yes, I totally agree with Russell that key rotation should be treated as something like a rewrite manifest action, and when the rewrite completes, the old files with old keys can be expired in a separated snapshot expiration action. Because of requirements like GDPR, this expiration would happen

Re: Key rotation in Iceberg data encryption

2021-03-25 Thread Russell Spitzer
I think you can treat the key rotation as a spark action like "RewriteManifestsAction" or something like that which creates a new Snapshot and new set of manifest files. If we want to be secure we would follow this up by immediately exporting and deleting previous snapshots and manifests. One probl