Yes, I totally agree with Russell that key rotation should be treated as something like a rewrite manifest action, and when the rewrite completes, the old files with old keys can be expired in a separated snapshot expiration action. Because of requirements like GDPR, this expiration would happen even if there is no one manually executing that expiration step. A flag can be added in the procedure to force expiring that snapshot during key rotation if needed. In the doc, I have designed this action as a call procedure to resemble its similarity with rewrite manifest.
I also agree that we should try to not use external files. For bloom filters, it might be fine because (1) the filter might be too large to fit in the manifest, and (2) once written we do not need to update it anymore. The encryption key is the complete opposite situation. As the stored procedure is executed distributedly anyway, I would not expect key rotation to cause too much burden on manifest rewriting. I have also worked in a system that stores encryption info in manifest, and I don’t see a scalability issue as long as the manifests are maintained properly to be of reasonable size. (if manifest is too large, all sorts of scalability issue would arise so it’s not just for the encryption key, and that’s why we have the rewrite manifest procedure at the first place) Based on the discussions so far, I provided the following updates to the doc: 1. Added concept Iceberg Encryption Key (IEK) as a generalized master key which can either be single-wrap (KEK ID) or double-wrap (MEK ID + encrypted KEK), just to make the rest of the doc cleaner without the repeated reference to single and double wrap. 2. Added concept of KeyResolver to tackle the problem of needing many KEKs for per partition, per day use cases, so that storing those keys in table metadata is scalable, and a mutable/replaceable medium can be plugged in if necessary. 3. In the key rotation procedure, added an option to allow users to (1) rotate KEK in single wrap, (2) rotate KEK in double wrap, (3) rotate MEK only in double wrap, so that the system is flexible enough to handle all of those cases. 4. In the key rotation procedure, added an option to force expiring the old table version. With these changes, I think we can avoid the question if we need to replace KEK or not for double wrapping, because the amount of work for 3.1 and 3.2 are mostly the same, with 3.2 just need to also rewrap the KEKs if MEK is rotated, but that only changes a single table metadata file. People who do not want to rotate KEK can just use option 3.3. Best, Jack Ye From: Russell Spitzer <russell.spit...@gmail.com> Reply-To: "dev@iceberg.apache.org" <dev@iceberg.apache.org> Date: Thursday, March 25, 2021 at 08:33 To: Iceberg Dev List <dev@iceberg.apache.org> Subject: RE: [EXTERNAL] Key rotation in Iceberg data encryption CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe. I think you can treat the key rotation as a spark action like "RewriteManifestsAction" or something like that which creates a new Snapshot and new set of manifest files. If we want to be secure we would follow this up by immediately exporting and deleting previous snapshots and manifests. One problem with this approach though is we basically lose the history of the table when we do this process. I'm not as big a fan of the new external file approach, mainly because I think we have a new object to keep track of and unlike bloom filters, this is extremely sensitive information. I think we could go this route but we should be very careful. Although this would probably let us update keys without losing history ...? On Thu, Mar 25, 2021 at 10:24 AM Gidon Gershinsky <gg5...@gmail.com<mailto:gg5...@gmail.com>> wrote: Hi all, We're working with Jack on a design for encryption of Iceberg data tables, and got a question / decision point we'd like to bring to the community's attention. Might be a bit exotic, but is important, so we have to try this. Any input on this subject, or pointers to relevant contacts / sources will be appreciated. A rather long text below; I tried to make it as short as possible to explain the question. We use the standard envelope encryption approach, where the data is encrypted with a "data encryption key" (DEK). There are lots of DEKs in a table, because we must generate a key per file/column (this is related to NIST requirements for cipher usage). Envelope encryption means that the many DEKs are encrypted with a few MEKs ("master encryption keys"). There could be just one MEK for the whole table, or for many tables; or a MEK per sensitive column. The MEKs are managed in a KMS ("key management service") - which stores them, and handles their access control. DEKs encrypted with MEKs are stored close to the data. Currently, in the "key_metadata" field in the Iceberg manifest files. Envelope encryption practice requires "key rotation", where the MEKs are replaced from time to time (in ~ weeks or months, as a precaution, or to limit the number of crypto operations as required by NIST; or after being compromised). MEK ID stays the same, but the key contents (and version) change. This means we have to delete all manifest files after the key rotation. Because they keep DEKs encrypted with the previous version of a MEK, which is not safe anymore. To avoid this - and to minimize Iceberg-KMS interactions (KMSs can be slow), we add a double envelope encryption mode (already in use in Parquet), where DEKs are encrypted with an intermediate KEK ("key encryption key"), which in turn is encrypted with a MEKs. There are less KEKs than DEKs (e.g. one KEK per writer process lifetime; or per day; or per N DEKs; or per partition; or per table; etc), but there are more KEKs than MEKs. The question is - upon MEK rotation, do we have to replace KEKs? If not, then we can keep DEKs encrypted with KEKs in the manifest files - which do not have to be deleted / replaced upon MEK rotation. KEKs encrypted with MEKs will be kept elsewhere, in a mutable/replaceable medium (which is easier, because there are much less KEKs than DEKs). If yes, then we either have to replace all manifest files in a table (once in a few weeks or months), or to keep "key_metadata" outside manifests, e.g. in new file types (like with the bloom filters). The size of the key_metadata entry - per data file - ranges from a few dozen bytes to a few dozen kilobytes. Cheers, Gidon
