Thanks Dan. I agree that clients cannot be required to process `format-version` before other fields. I don’t think the REST spec needs to ensure that old clients can load v4 tables. Old clients are not expected to support future table format versions. However, one of the incompatiblity is that the server should NOT return `200` with a response that violates its documented schema(e.g., missing the required manifest-list path in a snapshot) via v1 endpoint. That mighty be a reason for a v2 endpoint.
Based on that distinction, the following approach makes sense to me: 1. The v1 `loadTable` endpoint MUST NOT return v4 table metadata. Adding a `oneOf` alternative to the existing version of LoadTableResult would not solve the problem for already released clients. For the new response model, we could use `format-version` as a discriminator and guide clients to dispatch on it before validating version-specific fields. The v2 `loadTable` endpoint could then return metadata for v1–v4 tables. 2. If v1 `loadTable` is used for a v4 table, should the server return a documented `IcebergErrorResponse`, potentially a `406` with an error type such as `UnsupportedTableFormatVersionException`? As a side note: if we introduce a v2 endpoint, perhaps we should collect other requirements now to reduce the likelihood of another version bump in the near future. One example is partially loading table metadata so that catalogs do not need to return large snapshot histories, as discussed in the earlier proposal [1]. [1] https://lists.apache.org/thread/qocy0w2y1wk1775x2n68mt9vompcbhmc Yufei On Fri, Aug 21, 2026 at 10:30 PM Daniel Weeks <[email protected]> wrote: > Yufei, > > The compatibility issue is related to the conflict of relaxing the > required table location field for relative patha and table version field. > > We cannot guarantee which order a client processes the fields. Therefore, > if the missing location field is encountered, an older client would > reasonably fail due the missing required field and never have an > opportunity to perform the version check. > > There will be more fields potentially as we add/modify fields for the > other features introduced in v4, but this one alone highlights the > compatibility issue. > > -Dan > > On Thu, Aug 20, 2026, 3:10 PM Yufei Gu <[email protected]> wrote: > >> I largely agree with Dan and Amogh. I just want to clarify one point: >> >> > clients will fail to parse the JSON before they can determine it's a >> v4 table >> >> TableMetadata is required in LoadTableResult, and format-version is >> required within TableMetadata.[1] So, absent an incompatible change to the >> metadata’s JSON structure or a strict client-side parser, a client should >> be able to identify the table format version before interpreting the rest >> of the metadata. If the payload is malformed, failure is expected for v1 >> clients as well. That case alone does not seem like a compatibility concern. >> >> 1. >> https://github.com/apache/polaris/blob/4a3fba7d4b26ef9c985086d5aba53b26b0b63935/spec/iceberg-rest-catalog-open-api.yaml#L3541 >> >> Yufei >> >> >> On Tue, Aug 18, 2026 at 9:04 AM Daniel Weeks <[email protected]> wrote: >> >>> Hey Dhruv, >>> >>> I think the second approach is a much cleaner solution, though it relies >>> on some nuance for backward compatibility with clients. >>> >>> A few points to note: >>> 1) The REST spec's required/optional fields are not intended to match >>> exactly what is in the spec. TableMetadata is a great example of this, >>> where most fields are required in the table spec, but are optional in the >>> REST spec. >>> 2) Backward compatibility comes down to whether a client will break when >>> receiving a payload that it doesn't understand. >>> 3) Unfortunately, some of the gates you might expect to be possible >>> (like table version) aren't feasible if you break client-side parsers >>> (clients will fail to parse the JSON before they can determine it's a v4 >>> table). >>> >>> All that said, the only thing I believe we need to do is add a >>> `v2::loadTable` endpoint and require all v4 tables to be loaded through >>> that mechanism. It's important to note that only load table requires a >>> version bump since committing a table still use the normal snapshot update >>> and clients will know from the supported endpoints whether v4 is even >>> supported (via the v2::loadTable endpoint). >>> >>> I love to hear what others think, but I'm strongly in favor of both >>> starting the process of introducing new versioned endpoints and scoping >>> what we need for V4. >>> >>> There are a few open questions, like what other metadata changes will be >>> necessary (e.g. check constraints/default expressions/generated >>> columns/etc)? How do we want to introduce the new endpoint (e.g. allow it >>> to evolve with breaking changes until we vote on v2 endpoint / v4 spec? >>> >>> I think this is a great start though and would love to hear what others >>> think, >>> -Dan >>> >>> On Thu, Aug 13, 2026 at 3:58 PM Dhruv Arya <[email protected]> wrote: >>> >>>> Hi all, >>>> >>>> Kicking off this thread to discuss the initial REST spec changes needed >>>> for Iceberg V4: >>>> 1. root-manifest in Snapshot: We need the returned Snapshot to have a >>>> root-manifest . (oneOf of root-manifest or manifest-list). For V4 tables, >>>> manifest-list will not be set. >>>> 2. Add table-location: LoadTableResult, CommitTableResponse, and >>>> UnregisterTableResult all need to return table-location. When defined, this >>>> will be the absolute path of the table root. (Allows the paths inside the >>>> metadata to be relative making table migration easier). >>>> 3. Make metadata-location in CommitTableResponse optional: This is for >>>> catalog managed tables. >>>> >>>> Some of these changes are not backwards compatible: >>>> `manifest-list` in Snapshot is a required field right now. Some clients >>>> might rely on this field being set in the response. If we allow this to be >>>> empty when `root-manifest` is set, these clients could break. The same >>>> problem applies to `metadata-location`. Furthermore, many endpoints >>>> (unregister, table, etc) transitively depend on Snapshot. If we create a >>>> new SnapshotV2, many endpoints will need a version bump to V2. >>>> >>>> I have a PR that shows what this would roughly look like: (approach 1) >>>> https://github.com/apache/iceberg/pull/17543 . The diff is pretty big >>>> and we end up duplicating many classes and endpoints. >>>> >>>> Another way of solving this would be to make a required response field >>>> optional (manifest-list and metadata-location) without bumping the >>>> model/api versions. This might be a more practical approach assuming that >>>> most clients don't break this way. This PR shows how minimal these changes >>>> are: (approach 2) https://github.com/apache/iceberg/pull/17642 >>>> >>>> I lean toward the in-place approach, assuming few clients actually >>>> require these fields to be present. Does that seem reasonable, or are there >>>> compatibility concerns I'm missing? >>>> >>>> Regards, >>>> Dhruv Arya >>>> >>>
