Hi Hongyue, I'll quickly explain our use case here, but if you want an in-depth discussion of why we actually need this, look (and comment) here: https://lists.apache.org/thread.html/owfk09ll0n1crtfbl8jnyzmg8tm1dbrg.
Basically, with each append, we stuff our proprietary index into a statistics puffin file. Since there is currently no way to do manifests on these files, we have to read everything from the last snapshot's index, append new data, and write it into the new snapshot's puffin file. If another transaction won over our transaction, we need to re-read the new index state and write a brand new puffin file. The SnapshotAncestryValidator can indeed cover the validation part, but as far as I understand, it can't encapsulate the retry logic of writing a new statistics file. We can start the transaction from the beginning, but with a custom PendingUpdate we can reduce the re-try scope to just the puffin file while keeping the data files valid. We will probably have some other types of updates too because we abuse the table properties for some global info. At the end of the day, this is a philosophical discussion rather than a technical one. Opening this up will allow people to build on top of Iceberg easier, but it can also create incompatibilities between people's data, which is something that Iceberg became popular for. Some of these things would hopefully be merged upstream at some point though so it can really go both ways. Cheers, Filip On Saturday, October 10th, 2026 at 12:49 AM, Hongyue Zhang <[email protected]> wrote: > > Hi Dzeri, > > > > Thanks for raising this. I wrote the PR that introduced appendUpdate > > (#13631), happy to share some context. > > > > First question: can your conflict detection and resolution semantics be > > expressed with SnapshotAncestryValidator? Dan added that hook in the 1.11 > > release as public API on SnapshotUpdate. It runs after the metadata > > refresh, fire on every commit retry, including the transaction retry path. > > That would be the preferred approach if validation only needs to inspect > > the refreshed table state and reject. > > > > If a custom PendingUpdate is a must, some of what you described is already > > reachable, like the transaction's TableOperations can be exposed through > > `((HasTableOperations) txn.table()).operations()`. > > > > On making appendUpdate public, I got pushback on that in the past: the > > wiring of an update into a transaction should stay protected, since that is > > where cleanup and retry are handled. A lighter alternative to reflection is > > a split-package shim. One can have a small class declared in package > > org.apache.iceberg inside your own jar. > > > > Happy to learn more about your custom update and see if there is a > > generalizable way to extend this, beyond what RowDelta and OverwriteFiles > > can express today. > > > > Thanks, > > Hongyue Zhang
publickey - [email protected] - 0x5E7E90EC.asc
Description: application/pgp-keys
signature.asc
Description: OpenPGP digital signature
