The push-based mirroring highlighted by Ryan is a popular use case. Polaris has already implemented notification APIs to address this, and there have been several Iceberg community discussions surrounding it. If I recall correctly, we generally agreed that the notification API is beneficial per last discussion. Polaris supports various notification types( https://github.com/apache/polaris/blob/be343f1b484d2d79877978ff55f3ad36031dc4a5/spec/polaris-catalog-apis/notifications-api.yaml#L108). The "UPDATE" type provides the same functionality as "register table force". I believe using notification APIs, or a similar concept is the right approach for achieving push-based mirroring, as "register table force" on its own isn't sufficient. Here is a list of notification types used in Polaris for reference: - CREATE - UPDATE - DROP - VALIDATE
Yufei On Mon, Feb 10, 2025 at 4:49 PM Steve Zhang <hongyue_zh...@apple.com.invalid> wrote: > Thank you Russell and Ryan. > > Let me start to work on a new API to support force table registration in > catalog. > > Thanks, > Steve Zhang > > > > On Feb 10, 2025, at 4:29 PM, rdb...@gmail.com wrote: > > Yeah, it sounds like a "register table force" is the right concept here. I > think we want to make sure that table updates remain change-based as > the best practice in the REST API. But there are some irregular use cases > that justify having some mechanism to completely replace the state (like > push-based mirroring). I think it makes sense to revisit mirroring and this > use case and come up with a path forward. > > On Mon, Feb 10, 2025 at 3:12 PM Russell Spitzer <russell.spit...@gmail.com> > wrote: > >> I still would like a "register table" force" option >> >> On Mon, Feb 10, 2025 at 5:06 PM Steve Zhang >> <hongyue_zh...@apple.com.invalid> wrote: >> >>> Thank you Dan for your detailed reply. Based on your explanation, do you >>> think it would be worthwhile to support non-linear or complete metadata >>> replacements in the REST implementation? I am happy to contribute but might >>> need some guidance from the community on the best approach. >>> >>> For additional context, we explored into the workaround of using a >>> combination of dropping table and re-registering the table with concerns of >>> reading in between. There’s also an attempt to add a force option to the >>> register-table API (https://github.com/apache/iceberg/pull/5327), which >>> would allow for metadata swap on an existing table. However, it was >>> suggested that use TableOperations.commit(base, new) is preferred to >>> achieve atomicity. >>> >>> Thanks, >>> Steve Zhang >>> >>> >>> >>> On Feb 10, 2025, at 1:49 PM, Daniel Weeks <dwe...@apache.org> wrote: >>> >>> Hey Steve, >>> >>> I think the issue here is that you're using the commit api in table >>> operations to perform a non-incremental/linear change to the metadata. The >>> REST implementation is a little more strict in that it builds a set of >>> updates based on the mutations made to the metadata and the commit process >>> applies those changes. In this scenario, no changes have been made and the >>> call is attempting a complete replacement. >>> >>> The other implementations are just blindly swapping the location, so >>> while that operation does achieve the effect you're looking for, it's not >>> the right semantics for the commit. >>> >>> You might want to consider using the "register table" operation instead, >>> which takes the table identifier and location to perform this type of swap. >>> >>> -Dan >>> >>> On Fri, Feb 7, 2025 at 10:17 AM Steve Zhang >>> <hongyue_zh...@apple.com.invalid> wrote: >>> >>>> Hey Iceberg Experts: >>>> >>>> I am seeking assistance and insights regarding an issue we’ve >>>> encountered with RESTTableOperations and its inability to support on-demand >>>> table metadata swaps. We are currently adopting the REST-based catalog from >>>> Hive and have noticed a potential gap in the TableOperations.commit() >>>> API. Typically, we use the commit API to revert a table to a previously >>>> known state, as demonstrated below: >>>> >>>> String deisredMetadataPath = >>>> "/var/newdb/table/metadata/00003-579b23d1-4ca5-4acf-85ec-081e1699cb83.metadata.json"" >>>> ops.commit(ops.current(), TableMetadataParser.read(ops.io(), >>>> dedeisredMetadataPath)); >>>> >>>> However, this approach is no longer working with the REST-based >>>> catalog. I suspect that the issue may be related to how the update type is >>>> modeled in RESTTableOperations. I have shared a unit test that reproduces >>>> the problem on https://github.com/apache/iceberg/issues/12134, where >>>> it works on JDBC and in-memory catalogs, but not with RESTCatalog. >>>> >>>> Best Regards, >>>> Steve Zhang >>>> >>>> >>>> >>>> >>> >
