Hi Dmitri, all, Following up on the early thread and Dmitri’s review on the POC (PR #4912 <https://github.com/apache/polaris/pull/4912>): the POC explores a different path: stamp the key into the entity’s internalProperties in the same transaction as the create, so the key and the table commit atomically.
That closes the record-after-success gap for operations that leave a surviving entity (e.g. createTable). Trade-off: keys ride on the entity hot path (size / write amplification on busy tables), vs. a separate store where retention and size are independent. POC scope today is createTable only, behind polaris.idempotency.enabled. I’ve added microbenchmarks on the read/write cost. Posting here since this is a meaningful shift from the separate-store plan, would appreciate input on whether we should pursue this. Thanks, Huaxin On Tue, Jun 2, 2026 at 3:00 PM huaxin gao <[email protected]> wrote: > Hi Dmitri, > > Sounds reasonable, and yes, that matches the plan. > > The idempotency persistence will be its own set of Java interfaces (a > standalone IdempotencyStore SPI), separate from the metastore persistence > interfaces, and wired and configured independently. So in principle you > could pair a NoSQL metastore with a JDBC idempotency store. > > I also agree with the failure-mode framing. A retry can be confirmed as > "previously processed" only if both the metastore change and the > idempotency record committed. If the metastore commits but the > idempotency record is lost, that retry won't be idempotent. That's the > same record-after-success gap we discussed under Model B, and as you note > it exists even with a shared persistence layer, so decoupling doesn't > make it worse. I think it's a reasonable assumption, and I'll document it. > > I'll get the PR up. Thanks again for the thorough discussion! > > Huaxin > > On Tue, Jun 2, 2026 at 12:17 PM Dmitri Bourlatchkov <[email protected]> > wrote: > >> Hi Huaxin, >> >> That sounds like a good plan to me. Looking forward to a PR. >> >> I think we still have time to adjust the impl. in the PR if other points >> are raised during review in GH or by email. >> >> With this approach my main ask regarding Persistence impl. is (repeating >> my >> previous comments) to isolate the Idempotency persistence java interfaces >> from "meta store" persistence interfaces. Ideally one should be able to >> combine a NoSQL MetaStore with a JDBC Idempotency persistence (in theory). >> >> This will naturally create a situation when a request retry can be >> confirmed as "previuosly processed" only if the original MetaStore change >> and the related Idempotency record insert both committed successfully. I >> believe this is a reasonable assumption. >> >> If we commit the MetaStore change but lose the Idempotency record, the >> client will not be able to re-try idempotently. However, this failure mode >> is still possible even if the MetaStore and Idempotency share the same >> Persistence layer. >> >> Does this sound reasonable to you? >> >> Thanks, >> Dmitri. >> >> On Tue, Jun 2, 2026 at 3:05 PM huaxin gao <[email protected]> wrote: >> >> > Hi Dmitri, >> > >> > Thanks! I think we're converging. >> > >> > On 422: you're right. It isn't specified in the IRC spec. I'd added it >> > as a safeguard, not because the spec requires it, so I'm fine making it >> > best-effort. >> > >> > On performance: that's a fair point. For a fast-changing table, a late >> > retry would usually fail on stale update requirements anyway, so a long >> > retention window doesn't add much practical value there. So I agree we >> > can cap retention and size in practice without losing much. >> > >> > Since you're fine with a specialized persistence impl for idempotency >> > keys, I'd propose we go that route. It keeps keys off the entity hot >> path >> > and lets us tune retention independently of entity size, so we get the >> > simplicity without the write-amplification and cache cost on busy >> tables. >> > >> > I'll proceed with the separate idempotency persistence in the PR unless >> > others object. Thanks for working through this with me! >> > >> > Huaxin >> > >> > On Tue, Jun 2, 2026 at 11:12 AM Dmitri Bourlatchkov <[email protected]> >> > wrote: >> > >> > > Hi Huaxin, >> > > >> > > Re: Question 1 (422 responses). >> > > >> > > I do not actually see any mention of 422 responses WRT >> Idempotency-Key in >> > > the IRC spec. Did I miss it? >> > > >> > > So, if that behaviour is not strictly specified, Polaris can choose to >> > > produce those response it in any way that makes sense for the >> > > Polaris implementation, I think. >> > > >> > > Re: Performance. >> > > >> > > Yes, it is a valid concern. My previous suggestion for this approach >> was >> > > not a strict requirement, but a point for discussion. Let's see if >> other >> > > reviewers comment on this aspect too. >> > > >> > > Thinking about the quick-changing table use case, I believe the more >> > > frequent are the updates to a table the less retention period is >> required >> > > by the client. If a client takes too long to re-try and other clients >> > make >> > > many in-between updates, that client is unlikely to commit >> successfully >> > > anyway due to table metadata having evolved beyond its expectations. >> > > >> > > All in all, in practice it should be possible to cap the size of >> > > idempotency data, I think. This may be a bit at odds with the current >> IRC >> > > spec language regarding idempotency key retention, but it might still >> > work >> > > effectively. >> > > >> > > Also, UUID v7 has a time component. The server should be able to >> > recognize >> > > keys created outside the range of "recent" entries and flag those >> cases >> > > (e.g. in log) for the Polaris Admin user to note and take corrective >> > > actions. >> > > >> > > As for me, I'm fine with a specialized persistence impl. for >> idempotency >> > > keys too. >> > > >> > > Cheers, >> > > Dmitri. >> > > >> > > On Tue, Jun 2, 2026 at 1:14 PM huaxin gao <[email protected]> >> > wrote: >> > > >> > > > Hi Dmitri, >> > > > >> > > > You're right: DELETE is a weak case for the idempotency key, so I'll >> > > > drop that objection. >> > > > >> > > > Two reasons I agree. First, a client can handle DELETE failures on >> its >> > > > own: reload the table, see that it's gone, and stop. Second, your >> > > > identity point is the deeper one. The DELETE API uses the table >> name, >> > not >> > > > a physical table id. If a name is dropped and recreated, it points >> to a >> > > > different table over time, so the key can't give a clean guarantee >> for >> > > > DELETE anyway. >> > > > >> > > > So I'm fine scoping idempotency to operations that leave a surviving >> > > > entity: create, commit/update, register, rename. For those, storing >> the >> > > > key in the entity properties works, and the delete-storage problem >> goes >> > > > away. >> > > > >> > > > That leaves two things I'd want to settle before we pick >> > entity-property >> > > > storage over a separate store: >> > > > >> > > > 1. Key reuse (422). With keys in entity properties there's no global >> > > > (realm, key) index. If a client reuses one key for a different >> > > > resource, we can't detect it and return 422. Are we okay treating >> > that >> > > > as best-effort? >> > > > >> > > > 2. Performance on the hot path. I want to second Yufei's concern. >> The >> > > > number of keys per entity is roughly (write rate × retention >> > window), >> > > > and the cost concentrates on the busiest tables. Since entity >> > > > properties are serialized as one blob and rewritten on each >> update, >> > > > every commit rewrites all the stored keys, not just the new one, >> so >> > a >> > > > hot table pays growing write amplification on its commit path, >> plus >> > > > larger loads and a heavier cache footprint, even for plain table >> > > > loads. We can bound this with a tight retention cap, but that >> > directly >> > > > shrinks the idempotency window, which is the part clients >> actually >> > > > rely on. A separate store keeps this off the hot path and lets >> > > > retention be tuned independently of entity size. >> > > > >> > > > So my main questions are: are we okay with best-effort 422, and how >> do >> > we >> > > > want to handle the hot-path cost? If both have good answers, I agree >> > > > entity-property storage is the simpler choice. WDYT? >> > > > >> > > > Thanks, >> > > > Huaxin >> > > > >> > > > On Tue, Jun 2, 2026 at 6:59 AM Dmitri Bourlatchkov < >> [email protected]> >> > > > wrote: >> > > > >> > > > > Hi Huaxin, >> > > > > >> > > > > Good point about handling DELETE idempotently! >> > > > > >> > > > > However, I wonder whether it is a critical use case? >> > > > > >> > > > > Do you expect DELETE to benefit a lot from the Idempotency Key? >> > > > > >> > > > > I'd think it should be fairly straightforward for the client to >> > reload >> > > > the >> > > > > table to be deleted in case of failures, discover that it is gone, >> > and >> > > > not >> > > > > retry. WDYT? >> > > > > >> > > > > There's still the question of whether the client is deleting the >> > table >> > > it >> > > > > actually intends to delete. Another client could delete the >> current >> > > table >> > > > > and create a new table under the same name while the first client >> is >> > > > > "deliberating". The IRC API does not provide for unique table >> > > > > identification in DELETE operations, as far as I know. The >> operation >> > is >> > > > > invoked simply on the name, which can map to different physical >> > tables >> > > at >> > > > > different times. Adding Idempotency Keys does not help in this >> > > context, I >> > > > > think. >> > > > > >> > > > > Thanks, >> > > > > Dmitri. >> > > > > >> > > > > On Mon, Jun 1, 2026 at 9:59 PM huaxin gao <[email protected] >> > >> > > > wrote: >> > > > > >> > > > > > Hi Dmitri, >> > > > > > >> > > > > > I like the idea — the atomic key write closes the in-flight gap, >> > and >> > > it >> > > > > > avoids the Iceberg metadata and spec issues. Agreed too that >> losing >> > > > keys >> > > > > > on already-deleted entities is harmless. >> > > > > > >> > > > > > But I think the harder case is delete operations themselves. For >> > drop >> > > > > > table/view/namespace, the operation removes the entity, so >> there is >> > > no >> > > > > > surviving entity to hold the key. A retry of a successful drop >> > should >> > > > > > return an equivalent success, but with entity-property storage >> the >> > > key >> > > > > > has nowhere to live — so the retry would just see "not found" >> and >> > > > behave >> > > > > > differently. Where would a drop's key live in this model? >> > > > > > >> > > > > > Thanks, >> > > > > > Huaxin >> > > > > > >> > > > > > On Mon, Jun 1, 2026 at 6:13 PM Yufei Gu <[email protected]> >> > > wrote: >> > > > > > >> > > > > > > One concern I have with storing idempotency records as entity >> > > > > properties >> > > > > > is >> > > > > > > the potential performance impact. Over time, an entity could >> > have a >> > > > > large >> > > > > > > number of idempotency key/value pairs. That would increase the >> > > > entity's >> > > > > > > size, which may affect load, update, serialization, and >> caching >> > > costs >> > > > > for >> > > > > > > normal catalog operations, even when idempotency is not >> involved. >> > > Use >> > > > > > cases >> > > > > > > such as table loading and entity in-memory caching could be >> > > affected. >> > > > > > > Before moving in that direction, I think it would be useful to >> > > better >> > > > > > > understand and measure the performance implications. If the >> > entity >> > > > size >> > > > > > > growth turns out to be negligible in practice, the approach >> may >> > > still >> > > > > be >> > > > > > > attractive because of its transactional simplicity. >> > > > > > > >> > > > > > > Yufei >> > > > > > > >> > > > > > > >> > > > > > > On Mon, Jun 1, 2026 at 2:17 PM Dmitri Bourlatchkov < >> > > [email protected] >> > > > > >> > > > > > > wrote: >> > > > > > > >> > > > > > > > Hi Huaxin, >> > > > > > > > >> > > > > > > > How about storing idempotency keys in the Polaris Entity >> > > properties >> > > > > > (not >> > > > > > > > Iceberg metadata)? >> > > > > > > > >> > > > > > > > I understand that entities can be deleted thus discarding >> > > > previously >> > > > > > > > recorded keys, but based on the use cases discussed so far, >> it >> > > does >> > > > > not >> > > > > > > > look like deleted entities should be a functional concern. >> > > > > > > > >> > > > > > > > Storing idempotency keys inside the entity will ensure that >> > their >> > > > > > updates >> > > > > > > > are processed in the same logical change set as the entity >> > > changes >> > > > > from >> > > > > > > the >> > > > > > > > IRC request payload. >> > > > > > > > >> > > > > > > > This will ensure uniform operations across all Persistence >> > > > > > > implementations >> > > > > > > > and will not require any Idempotency-specific Persistence >> > > changes. >> > > > > > > > >> > > > > > > > WDYT? >> > > > > > > > >> > > > > > > > Thanks, >> > > > > > > > Dmitri. >> > > > > > > > >> > > > > > > > On Sun, May 31, 2026 at 2:35 PM huaxin gao < >> > > [email protected] >> > > > > >> > > > > > > wrote: >> > > > > > > > >> > > > > > > > > Hi Dmitri, Robert, >> > > > > > > > > >> > > > > > > > > Thanks both. >> > > > > > > > > >> > > > > > > > > Dmitri — I agree with both of your points. >> > > > > > > > > >> > > > > > > > > - Idempotency storage will stay separate from the >> > metastore. >> > > It >> > > > > > will >> > > > > > > > > be separate in code and in transactions. We make the >> > > > > idempotency >> > > > > > > > > decision before the handler runs, or after it commits >> — >> > > never >> > > > > > > inside >> > > > > > > > > the metastore transaction. >> > > > > > > > > - I'll document the assumption you raised. Model B is >> only >> > as >> > > > > > strict >> > > > > > > as >> > > > > > > > > the spec wants if the client builds the request so >> that >> > at >> > > > most >> > > > > > one >> > > > > > > > > try can commit (for example, update requirements). The >> > > > > catalog's >> > > > > > > > > optimistic concurrency makes sure of this. Model B >> just >> > > > records >> > > > > > the >> > > > > > > > > result on top of it. I'll say this clearly in the >> Polaris >> > > > docs. >> > > > > > > > > >> > > > > > > > > Robert — I see why the operation-id-in-metadata idea is >> > > > appealing. >> > > > > If >> > > > > > > we >> > > > > > > > > write the id inside the commit, it is atomic with the >> change. >> > > > That >> > > > > > > would >> > > > > > > > > close the in-flight gap for table and view operations. >> That >> > is >> > > a >> > > > > real >> > > > > > > > > plus. >> > > > > > > > > >> > > > > > > > > But I don't think we should put the idempotency key in >> table >> > > > > > metadata. >> > > > > > > > > Here is why: >> > > > > > > > > >> > > > > > > > > 1. It only works for table and view operations. It can't >> help >> > > > > > namespace >> > > > > > > > > operations, grants, or other writes. A separate store >> > > handles >> > > > > all >> > > > > > of >> > > > > > > > > them with one mechanism. >> > > > > > > > > >> > > > > > > > > 2. It mixes two concerns. Idempotency is a REST/catalog >> > > concern. >> > > > > > Table >> > > > > > > > > metadata should describe the table — schema, snapshots, >> > > > > > > partitioning, >> > > > > > > > > sort order. A per-request id is not table state. I'd >> > rather >> > > > not >> > > > > > mix >> > > > > > > > > the two. >> > > > > > > > > >> > > > > > > > > 3. It bloats the metadata. To support retries we'd have to >> > keep >> > > > > > > > > operation-ids with some retention/TTL. metadata.json is >> > > > > rewritten >> > > > > > on >> > > > > > > > > every commit and read on every table load. For tables >> with >> > > > many >> > > > > > > > > writes, this adds real cost. And every client and >> engine >> > > that >> > > > > > reads >> > > > > > > > > the table pays it, not just the idempotency path. >> > > > > > > > > >> > > > > > > > > 4. It doesn't match the spec. The Iceberg REST spec >> defines >> > > > > > idempotency >> > > > > > > > > at the protocol layer — an Idempotency-Key header with >> a >> > > > > > server-side >> > > > > > > > > contract. It does not store idempotency in table >> metadata. >> > > > > Putting >> > > > > > > an >> > > > > > > > > operation-id there would be a new mechanism that isn't >> in >> > > the >> > > > > spec >> > > > > > > > > today. So it's a change to how the spec >> > > > > > > > > works, and a cross-project change too. >> > > > > > > > > >> > > > > > > > > So I'd prefer to keep the record in a separate idempotency >> > > store. >> > > > > We >> > > > > > > > > accept the in-flight gap, but it is bounded. The catalog's >> > > > > optimistic >> > > > > > > > > concurrency stops a duplicate commit from landing. And >> once a >> > > > > record >> > > > > > > > > exists, retries replay cleanly. >> > > > > > > > > >> > > > > > > > > Thanks, >> > > > > > > > > Huaxin >> > > > > > > > > >> > > > > > > > > On Sat, May 30, 2026 at 3:15 AM Robert Stupp < >> [email protected] >> > > >> > > > > wrote: >> > > > > > > > > >> > > > > > > > > > Hi all, >> > > > > > > > > > >> > > > > > > > > > Thanks for the clarifications. Russell's explanation is >> > > > > especially >> > > > > > > > > useful. >> > > > > > > > > > I agree, ambiguous request outcomes, for example, >> timeouts >> > or >> > > > > > network >> > > > > > > > > > connections being reset, are hard to reason about. >> > > > > > > > > > >> > > > > > > > > > Clients often cannot reliably reconcile from the current >> > > state >> > > > > > alone >> > > > > > > > for >> > > > > > > > > > table/view state mutating operations. >> > > > > > > > > > >> > > > > > > > > > I wonder whether the idempotency key should be recorded >> in >> > > the >> > > > > > > > table/view >> > > > > > > > > > metadata as an "operation-id", with an explicit >> retention >> > > > > > guarantee, >> > > > > > > > > maybe >> > > > > > > > > > tied to a server-provided minimum TTL. >> > > > > > > > > > The approach could reduce or change the role of a >> separate >> > > > > > > > > > idempotency-record table and handling of it. >> > > > > > > > > > >> > > > > > > > > > Request handling could roughly look like this: >> > > > > > > > > > if the current history/metadata already contains that >> > > > > > > "operation-id", >> > > > > > > > > > return equivalent-enough response without re-running >> > the >> > > > > > > operation. >> > > > > > > > > > >> > > > > > > > > > try the committing operation: >> > > > > > > > > > if the commit succeeds: >> > > > > > > > > > record the "operation-id" in the table/view >> metadata, >> > and >> > > > > > > > > > return the successful response. >> > > > > > > > > > if the commit runs into a conflict: >> > > > > > > > > > re-check whether the current metadata/history >> contains >> > > that >> > > > > > > > > > "operation-id" >> > > > > > > > > > if so: >> > > > > > > > > > return equivalent-enough response. >> > > > > > > > > > otherwise: >> > > > > > > > > > return the conflict response. >> > > > > > > > > > >> > > > > > > > > > This is not perfect either and needs spec work, >> retention >> > > > rules, >> > > > > > and >> > > > > > > > may >> > > > > > > > > > only work for table and view operations. >> > > > > > > > > > >> > > > > > > > > > I mostly want to separate the questions: >> > > > > > > > > > 1. What guarantees do clients actually need after an >> > > ambiguous >> > > > > > > outcome? >> > > > > > > > > > 2. Where should the durable evidence for the guarantee >> > live? >> > > > > > > > > > >> > > > > > > > > > Robert >> > > > > > > > > > >> > > > > > > > > > On Sat, May 30, 2026 at 4:30 AM Dmitri Bourlatchkov < >> > > > > > > [email protected]> >> > > > > > > > > > wrote: >> > > > > > > > > > >> > > > > > > > > > > Hi Russell, >> > > > > > > > > > > >> > > > > > > > > > > Thanks for the information! It clarifies the use case >> a >> > lot >> > > > (at >> > > > > > > least >> > > > > > > > > for >> > > > > > > > > > > me :) >> > > > > > > > > > > >> > > > > > > > > > > In short, I'd say the main benefit is allowing >> clients to >> > > > avoid >> > > > > > > > > conflicts >> > > > > > > > > > > (409) on re-submitting changes that got committed by >> the >> > > > server >> > > > > > > > without >> > > > > > > > > > the >> > > > > > > > > > > client receiving confirmation of the success. >> > > > > > > > > > > >> > > > > > > > > > > I believe the Iceberg REST Catalog spec [1] is >> formally >> > > > > stricter >> > > > > > > than >> > > > > > > > > > Model >> > > > > > > > > > > B when it states "the server ensures no additional >> > effects >> > > > for >> > > > > > > > requests >> > > > > > > > > > > that carry the same Idempotency-Key". Since Model B >> > permits >> > > > > > request >> > > > > > > > > > > re-execution, the possibility of additional side >> effects >> > > > cannot >> > > > > > be >> > > > > > > > > ruled >> > > > > > > > > > > out completely based on the proposed server-side >> > algorithm >> > > > > alone. >> > > > > > > The >> > > > > > > > > > > server must assume that the client forms the (change) >> > > request >> > > > > in >> > > > > > > > such a >> > > > > > > > > > way >> > > > > > > > > > > that only one execution attempt can succeed (e.g. by >> > using >> > > > > > "update >> > > > > > > > > > > requirements"). This is also mentioned in comments on >> > the >> > > > doc >> > > > > > [2]. >> > > > > > > > > > > >> > > > > > > > > > > This is probably worth mentioning in the Polaris docs >> > > related >> > > > > to >> > > > > > > > > > > our Idempotency-Key implementation. >> > > > > > > > > > > >> > > > > > > > > > > Assuming this kind of cooperation on the client side, >> I >> > > > believe >> > > > > > > > Model B >> > > > > > > > > > can >> > > > > > > > > > > be considered compliant with the spec [1]. >> > > > > > > > > > > >> > > > > > > > > > > In anticipation of fresh implementation PRs for this >> > > feature, >> > > > > I'd >> > > > > > > > like >> > > > > > > > > to >> > > > > > > > > > > re-emphasize (IIRC I mentioned this before) that, I >> > think, >> > > we >> > > > > > > should >> > > > > > > > > > avoid >> > > > > > > > > > > coupling Idempotency persistence with MetaStore >> > persistence >> > > > > (both >> > > > > > > > > > code-wise >> > > > > > > > > > > and transaction-wise). Model B processes >> > > Idempotency-related >> > > > > data >> > > > > > > > > outside >> > > > > > > > > > > the original change request's execution scope. >> > Idempotency >> > > > > > > decisions >> > > > > > > > > are >> > > > > > > > > > > made either before the request starts executing or >> after >> > it >> > > > is >> > > > > > > > > committed >> > > > > > > > > > to >> > > > > > > > > > > the MetaStore. >> > > > > > > > > > > >> > > > > > > > > > > [1] >> > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> https://github.com/apache/polaris/blob/4e4eaf840bf71d431b13034b0dd6f338261d8e8b/spec/iceberg-rest-catalog-open-api.yaml#L2098 >> > > > > > > > > > > >> > > > > > > > > > > [2] >> > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> https://docs.google.com/document/d/1hqTejVyYXDpL5MJcVc7NyhCslKaGH82QoqMEcUYPvkE/edit?tab=t.0 >> > > > > > > > > > > >> > > > > > > > > > > Cheers, >> > > > > > > > > > > Dmitri. >> > > > > > > > > > > >> > > > > > > > > > > On Fri, May 29, 2026 at 8:26 PM Russell Spitzer < >> > > > > > > > > > [email protected] >> > > > > > > > > > > > >> > > > > > > > > > > wrote: >> > > > > > > > > > > >> > > > > > > > > > > > The problem with a client attempting to determine if >> > it’s >> > > > > > > > operations >> > > > > > > > > > > > succeeded via load table, and the reason all this >> work >> > > has >> > > > > > > > > proceeded, >> > > > > > > > > > is >> > > > > > > > > > > > that there is no way for a client to guaranteed >> path to >> > > > > > actually >> > > > > > > > > > > determine >> > > > > > > > > > > > if a commit occurred. There are too many legitimate >> > > > > mechanisms >> > > > > > to >> > > > > > > > > erase >> > > > > > > > > > > > history from an Iceberg table to guarantee an >> operation >> > > > > > occurred. >> > > > > > > > > > > > >> > > > > > > > > > > > For example, you could check if your snapshot >> exists in >> > > > > > snapshot >> > > > > > > > > > history >> > > > > > > > > > > > but this could have been erased by expire snapshots. >> > > > > > > > > > > > >> > > > > > > > > > > > Or you could check if the schema was modified >> according >> > > to >> > > > > your >> > > > > > > > > update, >> > > > > > > > > > > but >> > > > > > > > > > > > this too could have been undone by another >> operation. >> > > > Client >> > > > > A >> > > > > > > adds >> > > > > > > > > > > column >> > > > > > > > > > > > but gets time out, Client B removes the Column, >> Client >> > A >> > > > > > retries >> > > > > > > > and >> > > > > > > > > > adds >> > > > > > > > > > > > the column again. >> > > > > > > > > > > > >> > > > > > > > > > > > Because of this the Iceberg client usually just >> bails >> > out >> > > > to >> > > > > he >> > > > > > > > user >> > > > > > > > > > with >> > > > > > > > > > > > an exception if it doesn’t get an actual >> confirmation >> > > that >> > > > > the >> > > > > > > > commit >> > > > > > > > > > > > succeeded from the server. This leaves the “can I >> retry >> > > or >> > > > > not” >> > > > > > > as >> > > > > > > > an >> > > > > > > > > > > > exercise to the end user. >> > > > > > > > > > > > >> > > > > > > > > > > > In practice, actual Iceberg users work around this >> sort >> > > of >> > > > > > thing >> > > > > > > by >> > > > > > > > > > > adding >> > > > > > > > > > > > all sorts of custom metadata to hopefully persist >> > history >> > > > in >> > > > > > the >> > > > > > > > > table >> > > > > > > > > > > > itself in some way that can’t be touched by expire >> > > > snapshots, >> > > > > > but >> > > > > > > > > this >> > > > > > > > > > is >> > > > > > > > > > > > usually very fragile and also relies on all clients >> > > > behaving >> > > > > > > well. >> > > > > > > > > I’ve >> > > > > > > > > > > > seen folks use custom table properties for example >> > > > “batch-5: >> > > > > > > > > committed” >> > > > > > > > > > > > then manually have their own retry logic check >> whether >> > > this >> > > > > > > > property >> > > > > > > > > is >> > > > > > > > > > > > set. Then, of course, they also have to add a bunch >> > > custom >> > > > > > logic >> > > > > > > to >> > > > > > > > > > make >> > > > > > > > > > > > sure they clean up this state as well. >> > > > > > > > > > > > >> > > > > > > > > > > > This is why Iceberg added the Idempotency path in >> the >> > > first >> > > > > > > place, >> > > > > > > > it >> > > > > > > > > > > gives >> > > > > > > > > > > > us a guaranteed way for clients to retry in case of >> a >> > > > network >> > > > > > > issue >> > > > > > > > > or >> > > > > > > > > > > > catalog issue with a guarantee they will not do >> > duplicate >> > > > > work >> > > > > > be >> > > > > > > > > > > retrying. >> > > > > > > > > > > > With this in place the client can now cleanly retry >> > > (within >> > > > > the >> > > > > > > > > > > idempotency >> > > > > > > > > > > > window) the same operation over and over without >> > throwing >> > > > an >> > > > > > > > > exception >> > > > > > > > > > to >> > > > > > > > > > > > the end user. Only in a situation where the catalog >> > > cannot >> > > > > > > respond >> > > > > > > > > > over a >> > > > > > > > > > > > very long time will the user actually have to do >> some >> > > sort >> > > > of >> > > > > > > > > > > > reconciliation. You can look at the history of the >> > > Iceberg >> > > > > > > client’s >> > > > > > > > > > retry >> > > > > > > > > > > > behavior with ambiguous server side or network >> errors >> > to >> > > > see >> > > > > > how >> > > > > > > > this >> > > > > > > > > > has >> > > > > > > > > > > > been a problem in the past. >> > > > > > > > > > > > >> > > > > > > > > > > > On Fri, May 29, 2026 at 1:24 PM huaxin gao < >> > > > > > > [email protected] >> > > > > > > > > >> > > > > > > > > > > wrote: >> > > > > > > > > > > > >> > > > > > > > > > > > > Hi Robert, >> > > > > > > > > > > > > >> > > > > > > > > > > > > Thanks for your reply! >> > > > > > > > > > > > > >> > > > > > > > > > > > > You're right that Model B does not prevent >> duplicate >> > > > > > execution. >> > > > > > > > The >> > > > > > > > > > > > > record is written only after success. So if a >> client >> > > > times >> > > > > > out >> > > > > > > > > while >> > > > > > > > > > > the >> > > > > > > > > > > > > first request is still running, a retry can run >> the >> > > > handler >> > > > > > > > again. >> > > > > > > > > > > There >> > > > > > > > > > > > > is no record yet to stop it. So Model B is >> "remember >> > > and >> > > > > > > replay a >> > > > > > > > > > > > > successful result," not "run exactly once." >> > > > > > > > > > > > > >> > > > > > > > > > > > > On the trade-off: Model A gives a stronger >> guarantee, >> > > but >> > > > > it >> > > > > > > > needs >> > > > > > > > > > > > > reserve/heartbeat/purge state, which adds >> complexity >> > > and >> > > > > > > > overhead. >> > > > > > > > > > > Model >> > > > > > > > > > > > > B is simpler and cheaper. The window it leaves >> open >> > is >> > > > > small, >> > > > > > > > and a >> > > > > > > > > > > > > client only retries after a timeout, so racing >> first >> > > > > requests >> > > > > > > > > should >> > > > > > > > > > be >> > > > > > > > > > > > > rare in practice. Every design is a trade-off, >> and my >> > > > view >> > > > > is >> > > > > > > > that >> > > > > > > > > > > Model >> > > > > > > > > > > > > B is the right one here. >> > > > > > > > > > > > > >> > > > > > > > > > > > > It also helps to be clear about where >> duplicate-work >> > > > > > protection >> > > > > > > > > > really >> > > > > > > > > > > > > comes from. It comes from the catalog itself, not >> > from >> > > > > > > > idempotency. >> > > > > > > > > > The >> > > > > > > > > > > > > catalog uses optimistic concurrency. If wo first >> > > attempts >> > > > > > race, >> > > > > > > > at >> > > > > > > > > > most >> > > > > > > > > > > > > one commit wins and the other gets a 409. >> Idempotency >> > > > sits >> > > > > on >> > > > > > > top >> > > > > > > > > of >> > > > > > > > > > > > that. >> > > > > > > > > > > > > It does not replace it. >> > > > > > > > > > > > > >> > > > > > > > > > > > > So what does Model B add over "the client just >> calls >> > > > > > loadTable >> > > > > > > > and >> > > > > > > > > > > > > reconciles"? Two things that I think are real: >> > > > > > > > > > > > > >> > > > > > > > > > > > > 1. The 422 check. loadTable can tell a client >> that >> > a >> > > > > table >> > > > > > > > > exists. >> > > > > > > > > > It >> > > > > > > > > > > > > cannot tell the client that the table THEY >> > created >> > > > > with >> > > > > > > THIS >> > > > > > > > > key >> > > > > > > > > > > is >> > > > > > > > > > > > > the one that succeeded. The record binds the >> key >> > > to >> > > > > > > > > (principal, >> > > > > > > > > > > > > operation, resource). If the same key is >> reused >> > > for >> > > > a >> > > > > > > > > different >> > > > > > > > > > > > > request, the server returns 422. The client >> > cannot >> > > > > > detect >> > > > > > > > this >> > > > > > > > > > on >> > > > > > > > > > > > > its own. >> > > > > > > > > > > > > >> > > > > > > > > > > > > 2. One server-side behavior for all mutating >> ops. >> > > > > > > create-table >> > > > > > > > > > > happens >> > > > > > > > > > > > > to reconcile cleanly with loadTable. But the >> > point >> > > > of >> > > > > > the >> > > > > > > > > > > > > Idempotency-Key header is that the client >> should >> > > not >> > > > > > have >> > > > > > > to >> > > > > > > > > > write >> > > > > > > > > > > > > reconciliation logic for every operation. >> For a >> > > > known >> > > > > > key, >> > > > > > > > the >> > > > > > > > > > > > > server turns what would be a 409 into an >> > > equivalent >> > > > > 2xx >> > > > > > > > > replay. >> > > > > > > > > > > The >> > > > > > > > > > > > > client gets a clean success instead of an >> error >> > it >> > > > has >> > > > > > to >> > > > > > > > > > special- >> > > > > > > > > > > > > case. >> > > > > > > > > > > > > >> > > > > > > > > > > > > There is a third, weaker benefit: once a record >> > exists, >> > > > > > retries >> > > > > > > > > stop >> > > > > > > > > > > > > seeing flip-flopping results. But that only helps >> > > after a >> > > > > > > record >> > > > > > > > > > > exists, >> > > > > > > > > > > > > which is exactly the window you pointed out is >> > > > unprotected. >> > > > > > > > > > > > > >> > > > > > > > > > > > > So I'll correct my earlier wording. This is not >> > > > convergence >> > > > > > on >> > > > > > > > > > exactly- >> > > > > > > > > > > > > once idempotency. It is a narrower guarantee: >> replay >> > a >> > > > > > recorded >> > > > > > > > > > result, >> > > > > > > > > > > > > plus detect key misuse. It sits on top of the >> > catalog's >> > > > > > > existing >> > > > > > > > > > > > > concurrency control. The real question for the >> list >> > is >> > > > > > simple: >> > > > > > > is >> > > > > > > > > > that >> > > > > > > > > > > > > narrower guarantee worth shipping on its own? Or >> do >> > we >> > > > need >> > > > > > > Model >> > > > > > > > > A's >> > > > > > > > > > > > > in-flight protection to have a strong idempotency >> > > > > guarantee? >> > > > > > > > > > > > > >> > > > > > > > > > > > > My view is that the narrow version is worth it for >> > now: >> > > > > it's >> > > > > > > the >> > > > > > > > > > > > > behavior the spec asks for, the 422 check can't be >> > done >> > > > > > > > > client-side, >> > > > > > > > > > > and >> > > > > > > > > > > > > it's a small change we can strengthen toward >> Model A >> > > > later >> > > > > > if a >> > > > > > > > > real >> > > > > > > > > > > use >> > > > > > > > > > > > > case needs it. Happy to hear what others think. >> > > > > > > > > > > > > >> > > > > > > > > > > > > Best, >> > > > > > > > > > > > > Huaxin >> > > > > > > > > > > > > >> > > > > > > > > > > > > On Fri, May 29, 2026 at 7:36 AM Robert Stupp < >> > > > > [email protected] >> > > > > > > >> > > > > > > > > wrote: >> > > > > > > > > > > > > >> > > > > > > > > > > > > > Hi Huaxin, >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > Thanks for writing this up and moving the design >> > > > > discussion >> > > > > > > > back >> > > > > > > > > to >> > > > > > > > > > > > dev@ >> > > > > > > > > > > > > . >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > Since you’re asking before locking in the >> > > > > implementation, I >> > > > > > > > think >> > > > > > > > > > we >> > > > > > > > > > > > > should >> > > > > > > > > > > > > > clarify one point. >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > Model B is certainly simpler than the >> lease-based >> > > > > approach, >> > > > > > > but >> > > > > > > > > I’m >> > > > > > > > > > > not >> > > > > > > > > > > > > > sure I fully understand what problem it still >> > solves. >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > As I read it, if a client times out while the >> > > original >> > > > > > > request >> > > > > > > > is >> > > > > > > > > > > still >> > > > > > > > > > > > > > running, a retry with the same key may not see >> an >> > > > > > idempotency >> > > > > > > > > > record >> > > > > > > > > > > > yet >> > > > > > > > > > > > > > and could run the handler again. >> > > > > > > > > > > > > > So this feels less like preventing duplicate >> > > execution >> > > > > and >> > > > > > > more >> > > > > > > > > > like >> > > > > > > > > > > > > > remembering a successful result after the fact. >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > For the create-table case, couldn’t a client >> > achieve >> > > > > > roughly >> > > > > > > > the >> > > > > > > > > > same >> > > > > > > > > > > > > > recovery by calling loadTable after an ambiguous >> > > > timeout >> > > > > > and >> > > > > > > > > > > > reconciling >> > > > > > > > > > > > > > from there? >> > > > > > > > > > > > > > Since Model B also rebuilds the response from >> > current >> > > > > > catalog >> > > > > > > > > > state, >> > > > > > > > > > > > I’m >> > > > > > > > > > > > > > trying to understand what it gives us beyond >> that. >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > I’m not against simplifying the design, but I >> think >> > > we >> > > > > > should >> > > > > > > > be >> > > > > > > > > > > clear >> > > > > > > > > > > > > > about the narrower guarantee before calling this >> > > > > > convergence. >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > Best, >> > > > > > > > > > > > > > Robert >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > On Fri, May 29, 2026 at 12:29 AM huaxin gao < >> > > > > > > > > > [email protected]> >> > > > > > > > > > > > > > wrote: >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > > Hi all, >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > I've simplified the proposed design for >> > > > Idempotency-Key >> > > > > > > > support >> > > > > > > > > > in >> > > > > > > > > > > > > > Polaris >> > > > > > > > > > > > > > > (Iceberg REST spec — retries with the same key >> > must >> > > > not >> > > > > > > > produce >> > > > > > > > > > > > > > additional >> > > > > > > > > > > > > > > side effects), and I'd like a wider review >> before >> > > > > > updating >> > > > > > > > the >> > > > > > > > > > > > > > > implementation PR (#4269 < >> > > > > > > > > > > > https://github.com/apache/polaris/pull/4269 >> > > > > > > > > > > > > >). >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > What changed >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > - Before (Model A, lease-based): reserve an >> > > > > idempotency >> > > > > > > row >> > > > > > > > > > > before >> > > > > > > > > > > > > > doing >> > > > > > > > > > > > > > > work → IN_PROGRESS / heartbeat → finalize >> after. >> > > > > > > > > > > > > > > - After (Model B, optimistic commit): run >> the >> > > > handler >> > > > > > > > first → >> > > > > > > > > > > > record >> > > > > > > > > > > > > > only >> > > > > > > > > > > > > > > after a successful (2xx) outcome. The record >> > stores >> > > > > > > binding + >> > > > > > > > > > > status, >> > > > > > > > > > > > > not >> > > > > > > > > > > > > > > the HTTP response body. Retries with the same >> key >> > > > > > re-derive >> > > > > > > > an >> > > > > > > > > > > > > equivalent >> > > > > > > > > > > > > > > response from current catalog state >> > > > > > > > > > > > > > > instead of replaying a stored payload. >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > The design doc still compares Model A and >> Model B >> > > > > > > > side-by-side >> > > > > > > > > so >> > > > > > > > > > > the >> > > > > > > > > > > > > > > trade-offs are explicit. So far the discussion >> > has >> > > > been >> > > > > > > > leaning >> > > > > > > > > > > > toward >> > > > > > > > > > > > > > > Model B — mutating REST operations only, >> 2xx-only >> > > > > > > > persistence, >> > > > > > > > > no >> > > > > > > > > > > > > > > response-body storage, and the known >> > > > > > > > > > > > > > > trade-offs (e.g. concurrent first-request >> races; >> > > see >> > > > > the >> > > > > > > > NOTES >> > > > > > > > > > > > section >> > > > > > > > > > > > > in >> > > > > > > > > > > > > > > the doc). >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > Does this direction look right before we lock >> in >> > > the >> > > > > > > > > > > implementation? >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > Comments on the doc >> > > > > > > > > > > > > > > < >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> https://docs.google.com/document/d/1hqTejVyYXDpL5MJcVc7NyhCslKaGH82QoqMEcUYPvkE/edit?tab=t.0 >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > or replies on this thread both work. >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > Thanks, >> > > > > > > > > > > > > > > Huaxin >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> >
