Hi Gideon, please add me to the meeting. On Wed, Sep 1, 2021 at 11:34 AM Maya Anderson <maya.anderson....@gmail.com> wrote:
> Hi Gidon, > > I would like to join the meeting. > > Thanks, > Maya Anderson > > > ---------- Forwarded message --------- >> From: Gidon Gershinsky <gg5...@gmail.com> >> Date: Wed, Sep 1, 2021 at 9:11 PM >> Subject: Fwd: Data encryption in Iceberg >> To: <dev@iceberg.apache.org> >> >> >> Hi all, >> >> Per the sync this morning, we'll have a meeting on encryption-related >> efforts in Iceberg. Before we discuss the day/time options, let us know >> who's interested to join, please respond here or send a direct message to >> Ryan, Jack or myself. >> >> Cheers, Gidon >> >> >> ---------- Forwarded message --------- >> From: Gidon Gershinsky <gg5...@gmail.com> >> Date: Mon, Aug 30, 2021 at 5:57 PM >> Subject: Re: Data encryption in Iceberg >> To: <dev@iceberg.apache.org> >> >> >> Hi Jack, >> >> Thank you. We've been indeed busy with building the Iceberg data >> encryption code, since we have quite a demand for this functionality (with >> timeline requirements..). >> I've published an initial end-to-end implementation (PR 3053), comprised >> of a new code that handles the generation of data keys, and of the existing >> code (with some modifications) from the current PRs listed below (so this >> is a joint work, with contributions from both of us; I'm sure there are >> ways to recognize PR co-authorship :). >> >> As I mentioned, this is the simplest version (without double wrapping, >> column-specific master keys and two-tier key management). I got a prototype >> for these advanced data encryption features, but thought it might be best >> to start with an MVP - easier to digest by the community, and allows for a >> gradual layer-by-layer implementation. In my understanding, MVP can start >> without key rotation - because the latter has two parts, with the main one >> (key rotation in KMS) being totally transparent to Iceberg; the other part >> (re-wrapping of key_metadata and re-writing of manifest files and manifest >> lists) is required in threat models that cover a risk of master keys being >> compromised/leaked - so this is a less universal requirement and can be >> added post-MVP. But if you hold a different view on this, or need the >> second part of key rotation now, I'm sure this is doable; I just hope it >> won't slow down the MVP work. >> >> Having said that - there is a feature I believe would be a really good >> addition to the MVP. This is the encryption of manifests and manifest >> lists. I presume you refer to it in your mail. If you have an internal >> branch with its implementation - porting this to open source will be much >> appreciated. We need this capability (yes, the data is encrypted; but the >> stats are not.. which is not great, even if they actually are highly >> aggregated, a sort of a range mask). >> >> We can chat about this at the upcoming sync, but I support the suggestion >> to set up a more detailed discussion to align the encryption-related >> efforts. >> >> Cheers, Gidon >> >> >> On Sun, Aug 29, 2021 at 11:08 PM Jack Ye <yezhao...@gmail.com> wrote: >> >>> Hi Gidon and Huaxin, >>> >>> Thanks for continuing with the effort in Iceberg encryption support. I >>> did not get enough time to work on this area since the design discussion, >>> so far I only managed to add key metadata for manifest file, and there are >>> quite a few changes in our internal branch that I need to port to open >>> source. I will start to do it in the next few days. >>> >>> Regarding the design, I wonder if we should first start with defining >>> the actions API with a Spark implementation for file encryption key >>> rotation, and then discuss the user experience. >>> >>> In the original design document, I think we did not reach a consensus >>> with the community around the actual way to expose key rotation >>> functionalities. In Spark, we can either do it through DDL extension, or >>> implement it as a procedure. Given that this is a long-running distributed >>> procedure, my feeling is that the community will lean towards a procedure >>> call. >>> >>> We can continue with the discussion around this while first doing the >>> detailed implementation. Let's set up a discussion around this so that we >>> can align the efforts. >>> >>> Best, >>> Jack Ye >>> >>> >>> On Wed, Aug 25, 2021 at 4:19 AM Gidon Gershinsky <gg5...@gmail.com> >>> wrote: >>> >>>> Hi all, >>>> >>>> We have briefly discussed this subject in a June sync, with a >>>> decision to continue via the mailing list. >>>> There are a number of pull requests from Jack and myself that implement >>>> a set of disjoint elements from the high-level design >>>> <https://docs.google.com/document/d/1kkcjr9KrlB9QagRX3ToulG_Rf-65NMSlVANheDNzJq4/edit?usp=sharing>. >>>> Some low-level details, such as generation and propagation of data keys, >>>> are not covered in this document. >>>> I have created a short (and hopefully simple) doc >>>> >>>> https://docs.google.com/document/d/19O_qiQumz_66CdWLpw38GFJEsUpnNxXckP9rnYIQnCo/edit?usp=sharing >>>> that focuses on these details and describes the bottom-up approach to >>>> generation of data keys, encryption of data/delete files, and >>>> options/phases for optimization of key management. The scope of the >>>> document is intentionally narrow, and currently focuses on the minimal >>>> simplest option. Reviews are very welcome. Later, this doc will be merged >>>> in (or referenced from) the master design document. >>>> >>>> A PR with a basic encryption DDL has been sent recently by Huaxin, you >>>> can find it here <https://github.com/apache/iceberg/pull/3013>. Next >>>> week, I'll send a pull request with an implementation of the minimal >>>> encryption option. This pull request collects the basics from my PRs 2639, >>>> 2638, 2640 and Jack's PR 2443; adding the key generation and other code >>>> that creates an end-to-end implementation of the minimal design >>>> <https://docs.google.com/document/d/19O_qiQumz_66CdWLpw38GFJEsUpnNxXckP9rnYIQnCo/edit?usp=sharing>. >>>> This PR comes with an example proposed by Ryan - using a table encryption >>>> key from a keyfile ("pkcs12" format - the closest thing to the "pem" format >>>> for symmetric keys). >>>> Besides the minimal version, I have a draft implementation of more >>>> advanced data encryption options (including per-column keys, double >>>> wrapping and two-tier management - all described in the master design doc) >>>> - but let's take this one step at a time, starting with the simplest >>>> option. >>>> >>>> Cheers, Gidon >>>> >>> > > -- > Regards, > Maya > -- Best, Yufei `This is not a contribution`
