As a consumer of Iceberg metadata I think something like this might be helpful. We used approach #2 for adding partial Iceberg V2 support to BigQuery external tables, but this was more straightforward as we just had to detect the existence of delete files. With V3 we will have to be very confident that we can detect all of the unsupported features before we add support for any one of them.
That being said I don't think that will be *that* difficult. Would it be very hard for metadata producers to populate this? On Mon, Apr 14, 2025 at 8:48 AM Jean-Baptiste Onofré <j...@nanthrax.net> wrote: > Hi folks, > > I started to work on multi args transforms, and you probably saw > Fokko's proposal about the way to deal with source-id/source-ids to > ensure backward compatibility. > > While working on the changes on iceberg-core/iceberg-java, I'm > wondering if we should not introduce Iceberg Features on metadata. > Let me explain what I have in mind. > In Table Spec V3, we have new functionalities: new types (timestamp > nz, variant, ...), default values, row lineage, etc. > For readers/writers, there are two ways to know if functionalities are > available or not: > 1. Reading the table version spec (v2, v3) > 2. Reading if metadata contains some fields (for instance, regarding > multi args transforms, we have source-id / source-ids). > It means that we already have to "parse" the metadata and likely > implement "complex" logic. > > In addition of table spec version, I wonder if we should not introduce > Iceberg Features in metadata, clearly listing/describing the supported > features, decoupled from table spec version: > > "features": ["row_lineage","variant","default_value"] > > Reader/writer can just check the features to know how to behave. We > would like more flexible to support features, unbinding from the table > spec version. > > Afaik, Delta has something similar. > > Long term, it could be extended to Data File format API proposed by > Peter, e.g. some features related to data files (that would be a > different layer, but similar idea). > > Thoughts ? > > Regards > JB >
