Based on the current design of the IRC incremental scan API (which returns a set of files), it can only support append only case. But to me, incremental scan is really scanning the delta between two snapshots, so it should also include deleted and updated tuples. This would become very powerful building block for scenarios like MV refresh. But to support that it would have to return tuple set, instead of file set, and that would depart from the norms of IRC APIs.
So I think at this point, we would need to clarify that `plan` API only support append-only case for incremental scan. For true data scan, it needs a good discussion on how to support that from the API/spec level. On 2026/07/23 22:06:59 Alexander Bailey wrote: > Hi all, > > I'd like to open a discussion about whether incremental scan semantics > (specifically > incremental append scans) should be codified in the table spec, and if so, > what the contract should be. > > The gap: > > Today the spec is effectively silent on incremental scans. In > practice, the Java > IncrementalAppendScan is the de facto reference implementation, and everyone > else is following it: > > - PyIceberg -- implemented by following Java > github.com/apache/iceberg-python/pull/3512 > > - iceberg-cpp -- incremental scan API following Java: > github.com/apache/iceberg-cpp/pull/559 > > - iceberg-rust -- I have a PR up that, likewise, just mirrors Java > > github.com/apache/iceberg-rust/pull/2337 > > > So we have four implementations converging on behavior that lives only in one > codebase. That's fragile: any of Java's incidental choices become the > unwritten > contract, and there's no spec to point to when implementations disagree. > > There's a good precedent for what the fix could look like. The spec already > has > a "Point-in-Time Reads (Time Travel)" section (iceberg.apache.org/spec#… > <https://iceberg.apache.org/spec/#point-in-time-reads-time-travel>) > that specifies > read semantics in prose: which metadata to use to resolve the snapshot > (snapshot-log), how the two histories can diverge, and what to do when the > metadata is missing (raise an informative error). Codifying them would fill > a gap right next to something the spec already covers. > > Why this is now also a catalog concern, not just an engine concern? > > This used to be purely an engine-side matter. But the REST scan-planning > API has changed that. REST spec already exposes start-snapshot-id > ("Starting snapshot ID for an incremental scan (exclusive)") and > end-snapshot-id ("Ending snapshot ID for an incremental scan (inclusive). > Required when start-snapshot-id is specified."), and the planTableScan > description states that incremental scans are planned by passing both, and > that mixing incremental with point-in-time parameters is invalid. > > In other words, the REST spec already obligates a catalog to produce > an incremental > scan plan, but "correct" is defined only by whatever Java happens to do. > Two conformant catalogs could return different plans for the same request > and both claim REST compliance. It also means boundary conditions > (exclusive start / inclusive end) currently live only in the REST > documentation, not in the spec, so a non-REST engine has no shared source > of truth. > > The core principle for me is that the result of an incremental scan should not > depend on where it was planned. Whether an engine plans the scan > locally against > table metadata, or delegates to a catalog via planTableScan, it should get > the same set of files back. Today that's only true by coincidence because > every planner happens to copy Java. If the semantics aren't in the spec, > then server-side planning and client-side planning are free to diverge, and > the same query returns different data purely based on whether the catalog > implements the scan-plan API. That's a correctness property that shouldn't > be left to chance I think. > > > Some of the questions I think would need answering: > > If we do put this in the spec, these are the ambiguities I'd want pinned > down (drawn from Java's current behavior): > > - Range / traversal: is the scan the linear ancestry from end-snapshot-id back > to but excluding start-snapshot-id? What's required if start is not an > ancestor > of end? (Java validates ancestry and errors if the start is not anancestor > / parent-ancestor of the end.) > > - Non-append operations in the range: Java today skips non-append snapshots in > the range -- appendsBetween keeps only snapshots whose operation is "append" > and ignores the rest (replace/overwrite/delete). > > - Boundary semantics: exclusive-start / inclusive-end is currently > only asserted > in the REST spec should it be normative in the spec more generally? > > - Which schema? Java projects the incremental scan result into the > current table > schema. But the range spans multiple snapshots, so "the snapshot schema" > isn't well-defined the way it is for a point-in-time scan. > > I don't have a strong predetermined answer on whether this belongs in the > spec, > I mainly want to surface that it's currently implementation-defined,that > multiple implementations are already depending on it, and that the > REST scan-plan > API has quietly turned it into a conformance question for catalogs too. > > > Interested to hear what folks think. > > Thanks, > > Xander >
