Thanks Robert for the thoughtful note!
A generic JAX‑RS/Quarkus idempotency layer would be useful broadly, and
Quarkus’s distributed cache is a good building block. For Polaris, though,
we need a few things that go beyond caching or generic locking:
- No external lock service: we use an atomic “first‑writer‑wins” reserve
via a unique key in durable storage (single upsert), so exactly one node
owns a key; others see the existing row.
- Spec alignment: Iceberg chose key‑only semantics (no payload
fingerprinting). Safety comes from first‑acceptance plus binding
{operationType, resourceId, realm}; mismatched reuse -> 422; duplicates do
not re‑execute.
- Liveness and failover: heartbeat/lease while IN_PROGRESS and
reconciliation on stale owners (finalize‑gap/takeover) so duplicates don’t
block indefinitely and we avoid double execution.
- Durable replay: persist a minimal, equivalent response (not
in‑memory/TTL cache) and a clear status policy (finalize only 2xx/terminal
4xx; never 5xx).
Phase I: I’ll focus on implementing this in Polaris behind a small
storage‑agnostic SPI and wiring the flows.
Phase II: we can revisit extracting the core into a reusable JAX‑RS/Quarkus
module, but for now I’d like to keep the scope on shipping Polaris v1.
Thanks,
Huaxin
On Tue, Nov 25, 2025 at 11:18 PM Robert Stupp <[email protected]> wrote:
> Hi all,
>
> To build an idempotent service, it seems necessary to consider some
> things, naming a few:
> * distributed locking, resilient to failure scenarios
> * distributed caching
> * request fingerprinting
> * request failure scenarios
>
> I think a generic JAX-RS idempotency functionality would be beneficial
> not just for Polaris.
> I can imagine that the Quarkus project would be very interested in
> such a thing. For example, Quarkus already has functionality for
> distributed caching in place, which is a building block for idempotent
> responses.
> Have we considered joining forces with them and leveraging synergies?
>
> Robert
>
> On Wed, Nov 26, 2025 at 4:57 AM huaxin gao <[email protected]> wrote:
> >
> > Hi Dmitri,
> >
> > Thanks for the reply and the detailed comments in the proposal. You’re
> > right: the goal is to implement the recently approved Iceberg
> > Idempotency-Key spec, and we don’t plan any additional REST Catalog API
> > changes in Polaris. I’ve refocused the proposal on the server-side
> > implementation and agree we should land the REST Catalog work first, then
> > extend to the Management API.
> >
> > I addressed your inline comments and added a small, backend-agnostic
> > Idempotency Persistence API (reserve/load/heartbeat/finalize/purge) so it
> > works across all storage backends (Postgres first).
> >
> > On the async tasks framework: agreed — there are synergies. I’ll keep
> this
> > in mind and align the idempotency store semantics with the async tasks
> > model.
> > Best,
> > Huaxin
> >
> > On Tue, Nov 25, 2025 at 12:21 PM Dmitri Bourlatchkov <[email protected]>
> > wrote:
> >
> > > Hi Huaxin,
> > >
> > > Thanks for resuming this proposal!
> > >
> > > In general, I suppose the intention is to implement the recently
> approved
> > > Iceberg REST Catalog spec change for Idempotency Keys. With that in
> mind, I
> > > believe the Polaris proposal probably needs to be more focused on the
> > > server side implementation now that the API spec has been finalized. I
> do
> > > not think Polaris needs any other API changes in the REST Catalog on
> top of
> > > the Iceberg spec.
> > >
> > > I'd propose to deal with the REST Catalog API first and then extend to
> the
> > > Management API (for the sake of simplicity).
> > >
> > > I added some more specific comments in the doc, but overall, I believe
> we
> > > need to consider what needs to be changed in the java Persistence API
> in
> > > Polaris because the idempotency feature probably applies to all
> backends.
> > >
> > > Also, as I commented [1] in earlier emails about this proposal, I
> believe
> > > some synergies can be found with the async tasks framework [2]. The
> main
> > > point here is orchestrating request execution among a set of
> distributed
> > > server nodes.
> > >
> > > [1] https://lists.apache.org/thread/28hx9kl4qmm5sho8jxmjlt6t0cd0hn6d
> > >
> > > [2] https://lists.apache.org/thread/gg0kn89vmblmjgllxn7jkn8ky2k28f5l
> > >
> > > Cheers,
> > > Dmitri.
> > >
> > >
> > > On Sat, Nov 22, 2025 at 7:50 PM huaxin gao <[email protected]>
> wrote:
> > >
> > > > Hi all,
> > > > I would like to restart the discussion on Idempotency-Key support in
> > > > Polaris. This proposal focuses on Polaris server-side behavior and
> > > > implementation details, with the Iceberg spec as the baseline API
> > > contract.
> > > > Thanks for your review and feedback.
> > > >
> > > > Polaris Idempotency Key Proposal
> > > > <
> > > >
> > >
> https://docs.google.com/document/d/1ToMMziFIa7DNJ6CxR5RSEg1dgJSS1zFzZfbngDz-EeU/edit?tab=t.0#heading=h.ecn4cggb6uy7
> > > > >
> > > >
> > > > Iceberg Idempotency Key Proposal
> > > > <
> > > >
> > >
> https://docs.google.com/document/d/1WyiIk08JRe8AjWh63txIP4i2xcIUHYQWFrF_1CCS3uw/edit?tab=t.0#heading=h.jfecktgonj1i
> > > > >
> > > >
> > > > Best,
> > > > Huaxin
> > > >
> > >
>
