Re: Extending Apache Iceberg Encryption Module

2021-03-25 Thread Gidon Gershinsky
ingle and double wrapping > use case. > > > > -Jack > > > > > > *From: *Gidon Gershinsky > *Reply-To: *"dev@iceberg.apache.org" > *Date: *Wednesday, March 24, 2021 at 05:19 > *To: *"dev@iceberg.apache.org" > *Subject: *RE: [EXTERNAL]

Re: Extending Apache Iceberg Encryption Module

2021-03-24 Thread Ye, Jack
rting the single and double wrapping use case. -Jack From: Gidon Gershinsky Reply-To: "dev@iceberg.apache.org" Date: Wednesday, March 24, 2021 at 05:19 To: "dev@iceberg.apache.org" Subject: RE: [EXTERNAL] Extending Apache Iceberg Encryption Module CAUTION: This email origi

Re: Extending Apache Iceberg Encryption Module

2021-03-24 Thread Gidon Gershinsky
Sounds good, thanks. Responding to the points below: *"we can choose to store the encrypted DEKs inside the manifest or as a separated instruction file with a pointer in key_metadata, and there are tradeoffs for those approaches"* For the latter, we are running a similar mechanism in Parquet encr

Re: Extending Apache Iceberg Encryption Module

2021-03-23 Thread Jack Ye
Thanks for the feedback to the doc, we are also closely following the Parquet encryption work and would like to have that in Iceberg as soon as possible with the right architecture. Here are some brief thoughts for the points you mentioned in the email, I will add more details in the google doc:

Re: Extending Apache Iceberg Encryption Module

2021-03-23 Thread Gidon Gershinsky
Hi Jack, We're working on Parquet encryption, which is about to be released in the upcoming parquet-mr-1.12 version. Recently, we've started to look into its integration in Iceberg. It became immediately clear we need to take a wider view that covers other types of encryption in Iceberg (file stre

Extending Apache Iceberg Encryption Module

2021-03-22 Thread Jack Ye
Hi everyone, To continue the discussion in the last sync meeting about encryption in Iceberg, here is the document for a proposal: https://docs.google.com/document/d/1kkcjr9KrlB9QagRX3ToulG_Rf-65NMSlVANheDNzJq4/edit?usp=sharing Would be very appreciated for any feedback. Best, Jack Ye