Hi Dmitri, Good catch! This is a subtle but important issue. Thanks for raising it. I think we could handle it in a few ways:
1. Hold the event emission until the all multi-table commit succeeds, and buffering per-table events until persistence confirms success. 2. Include a transaction ID and status (e.g., pending, committed, aborted) in emitted events so consumers can filter accordingly. This will add burden to downstreams. I think we could figure out a way to filter out while persisting them, so that the real consumers won't see the events with aborted status. I'm not sure which way is better at this moment, we need to take a deep look to evaluate both. Yufei On Wed, Nov 5, 2025 at 8:06 AM Dmitri Bourlatchkov <[email protected]> wrote: > Hi All, > > I'd like to highlight an aspect of the current Events behaviour with > respect to multi-table commits that Christopher and I discovered today. The > issue existed since the first Events implementation, AFAIK, it just did not > come into focus until now, AFAIK. > > Consider a multi-table commit request [1]. IcebergCatalogHandler will run > each individual table through a separate commit operation on the base > catalog [2]. The base catalog will issue events for each table separately > [3][4]. However, the overall commit to Polaris Persistence may still fail, > e.g. due to concurrent updates [5]. > > Now, we can have a situation when both before/after events are delivered > for a table, but the actual change that triggered the events is _not_ > persisted, therefore does not exist in the current state of the Polaris > catalog. > > Thoughts? > > [1] > > https://github.com/apache/polaris/blob/f934443114251f85d18c9a51ed61fc49a500a61a/runtime/service/src/main/java/org/apache/polaris/service/catalog/iceberg/IcebergCatalogHandler.java#L973 > > [2] > > https://github.com/apache/polaris/blob/f934443114251f85d18c9a51ed61fc49a500a61a/runtime/service/src/main/java/org/apache/polaris/service/catalog/iceberg/IcebergCatalogHandler.java#L1051 > > [3] > > https://github.com/apache/polaris/blob/f934443114251f85d18c9a51ed61fc49a500a61a/runtime/service/src/main/java/org/apache/polaris/service/catalog/iceberg/IcebergCatalog.java#L1405 > > [4] > > https://github.com/apache/polaris/blob/f934443114251f85d18c9a51ed61fc49a500a61a/runtime/service/src/main/java/org/apache/polaris/service/catalog/iceberg/IcebergCatalog.java#L1558 > > [5] > > https://github.com/apache/polaris/blob/f934443114251f85d18c9a51ed61fc49a500a61a/runtime/service/src/main/java/org/apache/polaris/service/catalog/iceberg/IcebergCatalogHandler.java#L1058 > > Cheers, > Dmitri. >
