I must say I'm impressed with the level of constructiveness and
technical quality in this discussion, we're off to a good start in this
project.
*For POC, I think what you conclude is mostly correct, I am currently
implementing the encryption spec, general encrypted file stream with KMS
API, and I
Sounds good, lets continue with some discussions through the doc. For POC, I
think what you conclude is mostly correct, I am currently implementing the
encryption spec, general encrypted file stream with KMS API, and I would expect
the low level file encryption integration to take place separate
Sounds good, thanks.
Responding to the points below:
*"we can choose to store the encrypted DEKs inside the manifest or as a
separated instruction file with a pointer in key_metadata, and there are
tradeoffs for those approaches"*
For the latter, we are running a similar mechanism in Parquet encr
Thanks for the feedback to the doc, we are also closely following the
Parquet encryption work and would like to have that in Iceberg as soon as
possible with the right architecture. Here are some brief thoughts for the
points you mentioned in the email, I will add more details in the google
doc:
Hi Jack,
We're working on Parquet encryption, which is about to be released in the
upcoming parquet-mr-1.12 version. Recently, we've started to look into its
integration in Iceberg. It became immediately clear we need to take a wider
view that covers other types of encryption in Iceberg (file stre
