I'm not a big fan of this, I am currently a strong supporter of the V3 is V3 approach. This is one of the reasons we decided to make row-lineage mandatory, we want to avoid encouraging engines from selectively adopting requirements.
On Tue, Apr 15, 2025 at 1:42 PM Fokko Driesprong <fo...@apache.org> wrote: > Hey JB, > > Thanks for raising this. This would be another way of indicating (next to > the format version) what's supported. At first glance, I'm reluctant to add > this. For two reasons: > > 1. Because of the added complexity, both from a technical perspective, > and because it also might confuse downstream users, for example, an engine > does support Iceberg V3, but not variant type. > 2. As you indicated, this is similar to what Delta has. One issue that > they are experiencing is that the users expect that you should also be able > to disable features. For example, when you have row-lineage enabled, and > you want to read the table with an engine that does not support > row-lineage, there is an expectation to disable row-lineage. This is > different from what we support today with the format-version which only > allows upgrades (and not downgrades), this will also add a lot of > complexity to the codebase. > > Curious to learn what others think. > > Kind regards, > Fokko > > Op ma 14 apr 2025 om 19:56 schreef Brian Hulette <bhule...@apache.org>: > >> As a consumer of Iceberg metadata I think something like this might be >> helpful. We used approach #2 for adding partial Iceberg V2 support to >> BigQuery external tables, but this was more straightforward as we just had >> to detect the existence of delete files. With V3 we will have to be very >> confident that we can detect all of the unsupported features before we add >> support for any one of them. >> >> That being said I don't think that will be *that* difficult. Would it be >> very hard for metadata producers to populate this? >> >> On Mon, Apr 14, 2025 at 8:48 AM Jean-Baptiste Onofré <j...@nanthrax.net> >> wrote: >> >>> Hi folks, >>> >>> I started to work on multi args transforms, and you probably saw >>> Fokko's proposal about the way to deal with source-id/source-ids to >>> ensure backward compatibility. >>> >>> While working on the changes on iceberg-core/iceberg-java, I'm >>> wondering if we should not introduce Iceberg Features on metadata. >>> Let me explain what I have in mind. >>> In Table Spec V3, we have new functionalities: new types (timestamp >>> nz, variant, ...), default values, row lineage, etc. >>> For readers/writers, there are two ways to know if functionalities are >>> available or not: >>> 1. Reading the table version spec (v2, v3) >>> 2. Reading if metadata contains some fields (for instance, regarding >>> multi args transforms, we have source-id / source-ids). >>> It means that we already have to "parse" the metadata and likely >>> implement "complex" logic. >>> >>> In addition of table spec version, I wonder if we should not introduce >>> Iceberg Features in metadata, clearly listing/describing the supported >>> features, decoupled from table spec version: >>> >>> "features": ["row_lineage","variant","default_value"] >>> >>> Reader/writer can just check the features to know how to behave. We >>> would like more flexible to support features, unbinding from the table >>> spec version. >>> >>> Afaik, Delta has something similar. >>> >>> Long term, it could be extended to Data File format API proposed by >>> Peter, e.g. some features related to data files (that would be a >>> different layer, but similar idea). >>> >>> Thoughts ? >>> >>> Regards >>> JB >>> >>
