Hi Amogh and Dan, > ... only thing I believe we need to do is add a `v2::loadTable` endpoint and require all v4 tables to be loaded through that mechanism.
That makes sense. If we are adding a new v2 endpoint (only for loadTable), should we require clients to first parse the format version before parsing anything else? This would make future upgrades easier. Are there any technical blockers for that? > If I understand option 1 correctly, I don't even think it's an option when considering compatibility; I guess this is referring to option 1 from approach 2 here where we don't add a v2 endpoint. Approach 1 from the original email ( https://github.com/apache/iceberg/pull/17543) is even more conservative - it adds a bunch of new v2 endpoints and classes. This is because if we fork Snapshot to create a SnapshotV2 (to make manifest-list optional), we will transitively need to update many classes and endpoints that reference it. On Thu, Aug 20, 2026 at 2:50 PM Amogh Jahagirdar <[email protected]> wrote: > If I understand option 1 correctly, I don't even think it's an option when > considering compatibility; clients aren't guaranteed to distinguish how to > parse and interpret metadata coming over the wire based off the format > version. The Java implementation may happen to parse format version first > but that's not a requirement in the spec. Though it's a bit awkward since > there have been changes for v3 like new data types or new optional snapshot > fields going over the wire which arguably also are not backwards > compatible. It's obviously ideal to be able to use existing APIs as much as > possible but V4 is arguably such a leap that in my opinion that we should > not really continue doing the same thing here. > > I agree with Dan that option 2 with an explicit v2 endpoint that handles > v1-v4 tables is the best. With all the other potential aspects like > check-constraints, default expressions etc , the additional models going > over the wire have richer semantics and capabilities to the point where I > think it justifies having a new versioned API. It gives a cleaner > compatibiltiy story like Dan was saying since V4 tables can only be loaded > from that new API and indication of catalog support for V4 is just > indicated via the supported endpoints. I also think the burden for catalogs > is pretty negligible here, we're talking about a new endpoint for handling > a largely new metadata structure so that doesn't feel unreasonable to me; > the client burden is even more negligible imo, if we can understand v4 > metadata, how much of a lift is it to start making calls to loadTableV2. > > >There are a few open questions, like what other metadata changes will be > necessary (e.g. check constraints/default expressions/generated > columns/etc)? How do we want to introduce the new endpoint (e.g. allow it > to evolve with breaking changes until we vote on v2 endpoint / v4 spec? > > I think the answer to the first question just depends on where we draw the > line as to what we want to get into v4. But assuming all of those, and > assuming the barrier is largely just sufficient implementation to prove > those spec proposals, I think the way to do that is just iterate on them > until we vote. > > In my head at least, for a voting perspective, I think the v4 spec should > be voted on as a pre-requisite before voting on the v2 REST spec. I'm open > to alternatives here but a I imagine lot of the stuff going over the wire > for this API is for the most part contingent on what's also persisted so we > need consensus on the former first? > > Thanks, > Amogh Jahagirdar > > On Tue, Aug 18, 2026 at 10:04 AM Daniel Weeks <[email protected]> wrote: > >> Hey Dhruv, >> >> I think the second approach is a much cleaner solution, though it relies >> on some nuance for backward compatibility with clients. >> >> A few points to note: >> 1) The REST spec's required/optional fields are not intended to match >> exactly what is in the spec. TableMetadata is a great example of this, >> where most fields are required in the table spec, but are optional in the >> REST spec. >> 2) Backward compatibility comes down to whether a client will break when >> receiving a payload that it doesn't understand. >> 3) Unfortunately, some of the gates you might expect to be possible (like >> table version) aren't feasible if you break client-side parsers (clients >> will fail to parse the JSON before they can determine it's a v4 table). >> >> All that said, the only thing I believe we need to do is add a >> `v2::loadTable` endpoint and require all v4 tables to be loaded through >> that mechanism. It's important to note that only load table requires a >> version bump since committing a table still use the normal snapshot update >> and clients will know from the supported endpoints whether v4 is even >> supported (via the v2::loadTable endpoint). >> >> I love to hear what others think, but I'm strongly in favor of both >> starting the process of introducing new versioned endpoints and scoping >> what we need for V4. >> >> There are a few open questions, like what other metadata changes will be >> necessary (e.g. check constraints/default expressions/generated >> columns/etc)? How do we want to introduce the new endpoint (e.g. allow it >> to evolve with breaking changes until we vote on v2 endpoint / v4 spec? >> >> I think this is a great start though and would love to hear what others >> think, >> -Dan >> >> On Thu, Aug 13, 2026 at 3:58 PM Dhruv Arya <[email protected]> wrote: >> >>> Hi all, >>> >>> Kicking off this thread to discuss the initial REST spec changes needed >>> for Iceberg V4: >>> 1. root-manifest in Snapshot: We need the returned Snapshot to have a >>> root-manifest . (oneOf of root-manifest or manifest-list). For V4 tables, >>> manifest-list will not be set. >>> 2. Add table-location: LoadTableResult, CommitTableResponse, and >>> UnregisterTableResult all need to return table-location. When defined, this >>> will be the absolute path of the table root. (Allows the paths inside the >>> metadata to be relative making table migration easier). >>> 3. Make metadata-location in CommitTableResponse optional: This is for >>> catalog managed tables. >>> >>> Some of these changes are not backwards compatible: >>> `manifest-list` in Snapshot is a required field right now. Some clients >>> might rely on this field being set in the response. If we allow this to be >>> empty when `root-manifest` is set, these clients could break. The same >>> problem applies to `metadata-location`. Furthermore, many endpoints >>> (unregister, table, etc) transitively depend on Snapshot. If we create a >>> new SnapshotV2, many endpoints will need a version bump to V2. >>> >>> I have a PR that shows what this would roughly look like: (approach 1) >>> https://github.com/apache/iceberg/pull/17543 . The diff is pretty big >>> and we end up duplicating many classes and endpoints. >>> >>> Another way of solving this would be to make a required response field >>> optional (manifest-list and metadata-location) without bumping the >>> model/api versions. This might be a more practical approach assuming that >>> most clients don't break this way. This PR shows how minimal these changes >>> are: (approach 2) https://github.com/apache/iceberg/pull/17642 >>> >>> I lean toward the in-place approach, assuming few clients actually >>> require these fields to be present. Does that seem reasonable, or are there >>> compatibility concerns I'm missing? >>> >>> Regards, >>> Dhruv Arya >>> >>
