I still would like a "register table" force" option On Mon, Feb 10, 2025 at 5:06 PM Steve Zhang <hongyue_zh...@apple.com.invalid> wrote:
> Thank you Dan for your detailed reply. Based on your explanation, do you > think it would be worthwhile to support non-linear or complete metadata > replacements in the REST implementation? I am happy to contribute but might > need some guidance from the community on the best approach. > > For additional context, we explored into the workaround of using a > combination of dropping table and re-registering the table with concerns of > reading in between. There’s also an attempt to add a force option to the > register-table API (https://github.com/apache/iceberg/pull/5327), which > would allow for metadata swap on an existing table. However, it was > suggested that use TableOperations.commit(base, new) is preferred to > achieve atomicity. > > Thanks, > Steve Zhang > > > > On Feb 10, 2025, at 1:49 PM, Daniel Weeks <dwe...@apache.org> wrote: > > Hey Steve, > > I think the issue here is that you're using the commit api in table > operations to perform a non-incremental/linear change to the metadata. The > REST implementation is a little more strict in that it builds a set of > updates based on the mutations made to the metadata and the commit process > applies those changes. In this scenario, no changes have been made and the > call is attempting a complete replacement. > > The other implementations are just blindly swapping the location, so while > that operation does achieve the effect you're looking for, it's not the > right semantics for the commit. > > You might want to consider using the "register table" operation instead, > which takes the table identifier and location to perform this type of swap. > > -Dan > > On Fri, Feb 7, 2025 at 10:17 AM Steve Zhang > <hongyue_zh...@apple.com.invalid> wrote: > >> Hey Iceberg Experts: >> >> I am seeking assistance and insights regarding an issue we’ve >> encountered with RESTTableOperations and its inability to support on-demand >> table metadata swaps. We are currently adopting the REST-based catalog from >> Hive and have noticed a potential gap in the TableOperations.commit() >> API. Typically, we use the commit API to revert a table to a previously >> known state, as demonstrated below: >> >> String deisredMetadataPath = >> "/var/newdb/table/metadata/00003-579b23d1-4ca5-4acf-85ec-081e1699cb83.metadata.json"" >> ops.commit(ops.current(), TableMetadataParser.read(ops.io(), >> dedeisredMetadataPath)); >> >> However, this approach is no longer working with the REST-based >> catalog. I suspect that the issue may be related to how the update type is >> modeled in RESTTableOperations. I have shared a unit test that reproduces >> the problem on https://github.com/apache/iceberg/issues/12134, where it >> works on JDBC and in-memory catalogs, but not with RESTCatalog. >> >> Best Regards, >> Steve Zhang >> >> >> >> >
