This idea has been discussed before in several cases, see https://github.com/apache/incubator-iceberg/issues/16. We originally thought this would be the best way to support encryption metadata. However we instead made encryption a first-class concept in Iceberg.
The design premise we decided to follow is to avoid using arbitrary string tags in Iceberg. The reason for this is that it’s easy to use such custom metadata for features that should instead be embedded into the Iceberg spec itself. Or to put it another way, arbitrary metadata maps promote poor design around this untyped and unstructured field. Furthermore as these metadata tags grow large it will become less efficient to plan tasks as the entire metadata map has to be kept in memory, whereas in split planning where we have dedicated fields we can prune out the data file fields that aren’t needed for a given scan. -Matt Cheah From: Arvind Pruthi <[email protected]> Reply-To: "[email protected]" <[email protected]> Date: Tuesday, February 26, 2019 at 11:50 AM To: "[email protected]" <[email protected]>, Jacques Nadeau <[email protected]> Cc: Iceberg Dev List <[email protected]> Subject: Re: Question about replacing files and about Publishing Jars
smime.p7s
Description: S/MIME cryptographic signature
