Hi EJ, Thanks. Looking at the latest branch (4a5fef1c), PrimitiveCommitDependencyTest already covers the four cases from my test: threeTablePromotionIsAllOrNothing covers promoting every table and the "one table moved on main, nothing changes" case, and nativePromotionRacePublishesOneCompleteResult covers two concurrent promotions and a promotion racing a single-table updateTable. A PR of my test would duplicate them, so I won't send it.
For the PostgreSQL and CockroachDB runs, I'm happy to run those cases against both with the existing relational-jdbc test profiles and post the results here, if that helps. Regards, Ankur On Fri, Oct 30, 2026 11:17 AM, EJ Wang <[email protected]> wrote: > Hi Ankur, > > Thanks for testing the PoC against the multi-table promotion cases. A test > PR to the branch would be very helpful. I’d like to run the same cases > against PostgreSQL and CockroachDB as well, since H2 alone doesn’t > establish their concurrency behavior. > > Agreed on the duplicate-entity case. That’s an invalid request rather than > a transient conflict, so returning a retryable 409 is misleading. I’ll > check the appropriate Iceberg error mapping and cover it with a regression > test. > > On NoSQL, the longer-term goal is to reuse the same Polaris workflows > across backends, but I don’t want to assume the existing NoSQL > implementation can simply be migrated onto Primitives. We should validate > the required guarantees first, including dependency protection, > concurrency, and uncertain outcomes. The branch registry’s placement should > follow those requirements rather than dictate them. > > These tests and open questions will also be useful inputs to the Durable > design proposal. > > Thanks, > -ej > > On Fri, Oct 9, 2026 at 12:38 AM Ankur Kotwal <[email protected]> > wrote: > > > Hi EJ, all, > > > > I tried the PoC from the angle of the catalog-branches thread, since > > multi-table promotion depends on exactly this guarantee. > > > > I took the multi-table promotion test from #5740 (branch three tables, > > write > > to the branch, then move every table's main in one transactions/commit > that > > asserts main hasn't moved) and ran it on your branch with > > PrimitiveServiceTestFactory (H2), plus two race tests: > > > > - two promotions of the same tables from the same base, started together, > > 10 rounds: exactly one won every round, and the tables never showed a mix > > of the two; > > - a promotion racing a single-table commit to main of one of the tables, > > both asserting main is still at the base, 30 rounds with a random delay > so > > both orders occur (the promotion won 11, the writer 19): exactly one > > succeeded each time, and a failed promotion left every table unchanged. > > > > All four pass on your branch, and the same four pass on main with the > > in-memory metastore, so for this path the new layer keeps today's > > guarantees. As your README says, H2 doesn't tell us much about PostgreSQL > > or CockroachDB under serializable isolation; I didn't have a database to > > run them against here. If it's useful, I can send the test as a PR to > your > > branch so it runs in your PostgreSQL and CockroachDB jobs. > > > > Three things I noticed while reading the code: > > > > 1. updateEntitiesPropertiesIfNotChanged now validates the whole batch > > before > > writing, and rejects a batch that contains the same entity twice with > > TARGET_ENTITY_CONCURRENTLY_MODIFIED. Rejecting it makes sense, but in > > commitTransaction that status ends up as a CommitFailedException (409), > > which Iceberg clients treat as a conflict and retry, and the retry can > > never succeed. A distinct status, or rejecting duplicate identifiers > > with a 400 before the commit, might be clearer. > > > > 2. Running the location-overlap check over the whole batch inside the > same > > attempt closes the race between two requests that each see no overlap. > > I like that this lives once in the Manager rather than in each backend. > > > > 3. For catalog branches (the NoSQL named-pointer option in the other > > thread), the Primitives contract is enough as it stands: a branch head > > is one more key, and publishing is one attempt that reads the head and > > the target and commits the new pointer. Is the plan for the NoSQL > > persistence to stay on its own Persistence layer, or could it eventually > > sit on Primitives too? That would decide where the branch registry > > should live. > > > > Thanks for putting this together, > > Ankur > > > > On Wed, Oct 07, 2026 08:57 PM, Jean-Baptiste Onofré <[email protected]> > > wrote: > > > > > Hi guys, > > > > > > Thanks a lot EJ for the updated PoC and testing it across multiple > > > backends. > > > > > > At a high level, I really like the direction and separation of concerns > > > here: > > > - Having a business logic centralized in the shared Manager rathe than > > > re-implemented across backends avoids logic drift and fixes the > > maintenance > > > burden we have today. I remember it was my call when we had Community > > > Meetings were we discussed OpenJPA -> JDBC and later NoSQL. > > > - Keeping the storage adapter contract focused on basic storage > > operations > > > without "coupling" in table/catalog concepts is clean and portable (I > > have > > > the feeling it was discussed while back as well :)). > > > - Ensuring STS and object storage calls stay decoupled from the > database > > > transaction boundary is essential. > > > > > > I will take a new look on your branch, we probably need a couple of > weeks > > > or so (several of us are travelling to CoC). > > > > > > Personally, I'm interested in closely examining the failure handling > > around > > > uncertain commits. > > > > > > Thanks! > > > > > > Regards > > > JB > > > > > > On Thu, Oct 8, 2026 at 5:33 AM Dmitri Bourlatchkov <[email protected]> > > > wrote: > > > > > > > Hi EJ, > > > > > > > > Thanks for sharing the PoC! > > > > > > > > I'll try to explore that code, but cannot promise a quick turnaround > :) > > > > Reviewing this may take me a couple of weeks (due to other > priorities). > > > > > > > > Cheers, > > > > Dmitri. > > > > > > > > On Mon, Oct 5, 2026 at 4:47 PM EJ Wang < > [email protected] > > > > > > > wrote: > > > > > > > > > One detail I omitted: the PoC also includes FoundationDB and > Spanner > > > > > adapters alongside JDBC. They reuse the same business-aware > Manager. > > > The > > > > > adapters implement generic storage operations without table, > catalog > > or > > > > > grant rules. > > > > > > > > > > This exercises both goals: addressing consistency problems such as > > > > > concurrent creation at overlapping table locations, and reusing > that > > > > logic > > > > > across databases. The concurrency tests have run on FoundationDB, > > > > > PostgreSQL and CockroachDB. Spanner coverage currently uses the > > > emulator, > > > > > with three concurrency tests skipped. > > > > > > > > > > These are targeted feasibility checks. Production validation and > > > > > integration with Polaris’s existing NoSQL backend remain follow-up > > > work. > > > > > > > > > > -ej > > > > > > > > > > On Mon, Oct 5, 2026 at 1:36 PM EJ Wang < > > [email protected] > > > > > > > > > wrote: > > > > > > > > > > > Hi Dmitri, all, > > > > > > > > > > > > Two requests can both check for overlapping table locations, see > > > none, > > > > > and > > > > > > then create conflicting tables. We need the checks and the > > resulting > > > > > writes > > > > > > to be protected together. > > > > > > > > > > > > My proposal is to implement these rules once in shared Polaris > code > > > > (the > > > > > > Manager). This business-aware layer checks them in the > transaction > > > that > > > > > > creates the table. Database adapters would implement a > > > > business-agnostic > > > > > > interface (Primitives) for generic reads, writes and commit. They > > > would > > > > > not > > > > > > implement table, catalog or grant rules, so adding a database > > reuses > > > > the > > > > > > same Polaris behavior. > > > > > > > > > > > > This PoC assumes the database itself can protect the relevant > reads > > > and > > > > > > writes in one serializable transaction. Calls to object storage > or > > > > > > credential services happen outside it. > > > > > > > > > > > > The Java PoC > > > > > > < > > > > > > > > > https://github.com/flyingImer/polaris/tree/prototype/durable > > > -java-primitives/persistence/primitives > > > > > > > > > > > > reuses the existing transactional Manager and tests selected > races, > > > > > > authorization retries and uncertain commits. It uses a temporary > > data > > > > > > layout and reads more data than we'd want in production. Existing > > > NoSQL > > > > > > integration, production tuning and migration remain follow-up > work. > > > > > > > > > > > > I'm not proposing an additional request-wide Persistence Session > > yet. > > > > > > Existing business-aware request code still owns authorization, > > > external > > > > > > work and retries. > > > > > > > > > > > > The PoC allows an already-authorized metadata change to finish. A > > > fresh > > > > > > request retry checks authorization again. An uncertain commit > stops > > > > > > automatic retries and retains files that may already be > referenced. > > > > These > > > > > > policies still need agreement. > > > > > > > > > > > > Would this separation be a useful starting point? > > > > > > > > > > > > -ej > > > > > > > > > > > > > > > > > > On Tue, 22 Sep 2026 19:08:49 -0400, Dmitri Bourlatchkov < > > > > > [email protected]> > > > > > > wrote: > > > > > > > > > > > > Hi EJ, > 1. Which backend assumptions should we design for? > [...] I > > > > know > > > > > I > > > > > > used the term "transaction" a lot during the call, but the real > > > > challenge > > > > > > is about establishing an SPI with clear contracts that could be > > > > > implemented > > > > > > by JDBC and NoSQL persistence using the database features natural > > to > > > > each > > > > > > model. I am fairly confident that a JDBC impl. is possible with > > > > > > Serializable transaction. We should be able to solve all the > > problems > > > > > > raised in this thread. However, that will make in-memory caching > > > > > impossible > > > > > > because the database needs to see all the reads (and writes) in a > > Tx > > > to > > > > > be > > > > > > able to ensure serializable guarantees. If a cache prevents a > read > > > from > > > > > > hitting the database, that read (e.g. storage config) will not > be a > > > > > factor > > > > > > in serializability. This connects to my earlier question about > the > > > > > > in-memory cache and JDBC (title: Usefulness of > InMemoryEntityCache) > > > Are > > > > > > different approaches possible with JDBC? Probably. What is > > involved? > > > > This > > > > > > depends on the SPI contract, I think. > 2. How do we protect the > > data > > > > > used > > > > > > by validation? [...] The point mentioned by Robert previously > IIRC, > > > is > > > > > that > > > > > > we need to ground all runtime decisions to a particular state of > > the > > > > > > catalog. If at commit time the state of the catalog in the > database > > > > > shifts > > > > > > from what was used during validation, the request has to restart > > from > > > > > > scratch. Current JDBC Persistence does this only for > relationships > > > > > between > > > > > > entities and grants and between versions of the same entity. > > > > > Relationships > > > > > > between table locations and storage config are not actualized for > > the > > > > > > purpose of state consistency tracking. From my POV we need to > begin > > > by > > > > > > formulating this Persistence state in the Persistence SPI terms. > > > Then, > > > > > > think how we can implement that for JDBC. NoSQL Persistence > already > > > > has a > > > > > > per-catalog state (not granular), so wiring the new SPI to NoSQL > > > should > > > > > not > > > > > > be too hard, I hope. The state in the SPI does not have to be > > global > > > to > > > > > the > > > > > > catalog (as in NoSQL), but it needs to be able to express the > idea > > of > > > > > which > > > > > > elements were engaged during request processing so serve as input > > to > > > > > > Persistence impl. for cosistency guarantees. If Persistence > caches > > > data > > > > > in > > > > > > memory (for example), it will have to do so in a way that does > not > > > > > violate > > > > > > consistency expectations. From my POV we could begin by defining > a > > > > > > "Persistence Session" concept. Each request is 1:1 with a > session. > > > All > > > > > > reads and writes go through the session. The next step would be > the > > > > JDBC > > > > > > impl., which probably needs some brainstorming... but I wonder if > > we > > > > > could > > > > > > reach a consensus at a high level first. WDYT? Thanks, Dmitri. On > > > Tue, > > > > > Sep > > > > > > 22, 2026 at 5:55 PM EJ Wang <[email protected]> > > wrote: > > > > > > > > > > Thanks for the recap, Dmitri. I'd like to follow up on the three > > > points > > > > > I > > > > > > > raised during the sync. We didn't have much time to discuss them, > > so > > > > it > > > > > > > would help to confirm these before choosing an SPI design. > > 1. > > > Which > > > > > > backend assumptions should we design for? > > Within one Polaris > > > server > > > > > > instance, do we assume entities, grants and > secrets all use the > > > same > > > > > > database backend, or must we support mixing > backends across > > them? > > > > > > Must > > > > > > every supported persistence implementation provide transactional > > > > > > > > guarantees? Does the database itself need to provide > transactions, > > or > > > > > can > > > > > > > Polaris build that capability on top of simpler database > > operations? > > > > > > > > > > > > > > > I'm happy to work with a homogeneous assumption if that's the > > > intended > > > > > > > > > > > scope. Existing NoSQL illustrates the second question: it can > > > publish > > > > > > > changes to multiple objects under one reference using a single > > > > > conditional > > > > > > > pointer update. > > 2. How do we protect the data used by > > > validation? > > > > > > > > > > > > > Your storage-configuration and location-overlap examples make the > > > > > problem > > > > > > > concrete. Consider these races: > > A table creation passes > > > validation > > > > > > against the catalog's allowed locations, > then another request > > > changes > > > > > > that configuration before the table is > committed. > Two > > > > table-creation > > > > > > requests each check for overlapping locations, both see > no > > overlap, > > > > and > > > > > > then create tables at conflicting locations. > > Checking only > the > > > > record > > > > > > being written cannot cover these dependencies. The > first also > > > depends > > > > > on > > > > > > the catalog configuration. The second depends on the > absence of > > > > another > > > > > > overlapping table. > > Could the persistence layer protect these > > > reads > > > > > > together with the writes, > using backend-internal > versions/tokens > > or > > > > > > database isolation? Shared code > would still need to identify > the > > > > > required > > > > > > reads and checks. Could we keep > the mechanism for protecting > them > > > > > inside > > > > > > each backend? > > 3. Business rules are duplicated across backend > > > > > > implementations > > Today, the TreeMap > < > > > > > > > > > > > > > > > > https://github.com/apache/polaris/blob/5de0c6a900a7fad77e7d0 > > > 663222ce2ee23621917/polaris-core/src/main/java/org/apache/ > > > polaris/core/persistence/transactional/TreeMapTransactionalP > > > ersistenceImpl.java#L446-L481 > > > > > > > > > and JDBC > < > > > > > > > > > > > > > > > > https://github.com/apache/polaris/blob/5de0c6a900a7fad77e7d0 > > > 663222ce2ee23621917/persistence/relational-jdbc/src/main/ > java/org/apache/ > > > polaris/persistence/relational/jdbc/JdbcBasePersis > > > tenceImpl.java#L1002-L1046 > > > > > > > > > implementations each load a secret, check its principal, > > apply > > > > the > > > > > > > > > > > > rotation/reset steps and write it back. > > Changing these > business > > > > rules > > > > > > therefore requires keeping multiple > implementations in sync. A > > fix > > > > > > applied to one backend can be missed in > another, causing the > same > > > > > Polaris > > > > > > operation to behave differently depending > on the database. > > > My > > > > > > proposal is to implement each business workflow once, against a > > > common > > > > > > > > > > > set of storage operations with defined consistency and failure > > > > > semantics. > > > > > > > The shared workflow should not need to know whether those > > operations > > > > are > > > > > > > > > > > > implemented through JDBC transactions, NoSQL reference CAS or > > > another > > > > > > > mechanism. Each backend would implement the same storage > contract, > > > > > keeping > > > > > > > its execution details internal. A new database backend could > then > > > > reuse > > > > > > the > existing business workflows. > > Do these seem like > > reasonable > > > > > > starting points? We can then evaluate the SPI > against the > > original > > > > > > failure cases on both JDBC and existing NoSQL. > > -ej > > On > Fri, > > > Sep > > > > > 18, > > > > > > 2026 at 12:10 PM Dmitri Bourlatchkov <[email protected]> > > wrote: > > > > > > > > > > > > Side note: I just came across [5541], which shows a case where a > > > failed > > > > > > > > > > > > > request leaves persisted side effects. > > > > This can be seen > as > > a > > > > > coding > > > > > > mistake, of course. However, this is about > > executing one > > request > > > > as a > > > > > > single, atomic unit. This is something that > > probably needs > > > > > foundational > > > > > > support and coding patterns in Polaris code. > > > > Cheers, > > > > > > Dmitri. > > > > > > > > > > > > > [5541] https://github.com/apache/polaris/pull/5541 > > > > On > > Fri, > > > > Sep > > > > > > 18, 2026 at 3:02 PM Dmitri Bourlatchkov <[email protected]> > > > > > wrote: > > > > > > > > > > > > > > > > > > > > Hi All, > > > > > > Here's a recap of the Community Sync > > > > discussion > > > > > > as I remember it. I'm > > sure > > > my recollection is not > > complete, > > > > so > > > > > > please add your thoughts to this > > thread. > > > > > > * > Polaris > > > > > > traditionally relies on entity version number to ensure > > > > > > > consistency > > > > > > of changes. > > > > > > This works well for internal RBAC grants, > > > which > > > > > > also have version > numbers > > > cross-referenced at the > > Persistence > > > > > > layer. > > > > > > I do not think this works in more general > cases > > > like > > > > > > validating > locations > > > wrt catalog-level storage > > > configuration. > > > > > > > > > > > > > > > > > Another difficult case involves location overlaps between > > > > > concurrent > > > > > > > > table > > > creations. > > > > > > * We talked about how > > > > Persistence > > > > > > calls related to RDBMS transactions > (in > > > the JDBC > > persistence > > > > > case). > > > > > > > > > > > > There were some concerns raised about whether we need > > to > > > > > bind > > > > > > > Persistence > > > API to transactions explicitly. > > > > > > I > > think > > > > it > > > > > is > > > > > > a valid concern. At the same time, the Persistence API > > > > should > > > be > > > > > > clear about consistency guarantees across all backend > > > > > > > > > implementations. I think this needs more attention now that the > > > matter > > > > > > > > > > of > > > > > > > > > milti-entioty changes became prominent in [5035]. > > > > > > > > I > > > > > > personally believe that RDBMS-based Persistence in Polaris has to > > > use > > > > > > > > > > > > > > Serializable transaction isolation. Consequently, the core code > > > needs > > > > > > to > > > > > > > > > provide mechanisms to allow the Persistence implementation > to > > > > > > properly > > > connect reads and writes from API requests to JDBC > > > > > > transactions... > > whether > > > we use "transation" as an > > explicit > > > > > terms > > > > > > in the SPI or not. > > > > > > Note that currently, the same API > > > > request > > > > > > performs reads and writes in > > > _separate_ RDBMS transactions > > > (even > > > > > > separate connections). > > > > > > * We did not discuss > Persistence > > > SPI > > > > > > implications in the call, but > > > currently each Persistence > call > > > is > > > > > > assumed to be one distinct and > atomic > > > change (please > > correct > > > me > > > > > if > > > > > > I'm wrong). > > > > > > Consequently, supporting new use cases > > > involves > > > > > > introducing new SPI > > > methods with many parameters. This can > be > > > > seen > > > > > in > > > > > > PR [5035]. This tends > > to > > > overcomplicate the SPI. > > > > > > > > > > > > > > > > > > Ideally, I think the SPI should be simplified to deal with > > > multi-entity > > > > > > > > > > > > > > and single-entity changes in a coherent manner to avoid any > > > ambiguity > > > > > > > > > > > > > about > > > which method the caller should use. > > > > > > > [5035] > > > > > > https://github.com/apache/polaris/pull/5035 > > > > > > Cheers, > > > > > > > > > > > > > > > Dmitri. > > > > > > On Tue, Sep 15, 2026 at 11:06 AM Dmitri > > > > Bourlatchkov > > > > > < > > > > > > [email protected] > > > > > wrote: > > > > > >> Heads up: This > > > > discussion > > > > > > is on the Community Sync call agenda for > Sept > > >> 17. > > >> > > > > > > > > >> > > > > > > Interested parties, please plan to attend, if possible :) > > >> > > > > > > > > >> > > > > > > Cheers, > > >> Dmitri. > > >> > > >> On Wed, Sep 9, 2026 at > 1:48 PM > > > > > Dmitri > > > > > > Bourlatchkov <[email protected]> > > >> wrote: > > >> > > >>> Hi > > JB, > > > > > > > > > > > > > > > >>> > > >>> I've added an agenda item to the next community sync > > call > > > > for > > > > > > this > > (Sept > > >>> 17, 2026). > > >>> > > >>> Re: key points: > > did > > > > you > > > > > > mean a real GH discussion or dev email (this > > >>> thread)? > Just > > > > double > > > > > > checking :) I'm fine with either approach. > > >>> > > >>> > Cheers, > > > > > > > > > > > > >>> > > > > > > Dmitri. > > >>> > > >>> On Wed, Sep 9, 2026 at 12:54 PM > > Jean-Baptiste > > > > > > Onofré < > [email protected]> > > >>> wrote: > > >>> > > >>>> Hi > > > > Dmitri, > > > > > > > > >>>> > > >>>> I agree on the need to align on these points, > > though > > > > I'm > > > > > > not > entirely > > >>>> sure a dedicated meeting is necessary. > > Let's > > > > > start > > > > > > by using some > time > > >>>> during the next community meeting > to > > > > > discuss > > > > > > it. > > >>>> > > >>>> In the meantime, what if we outline the key > > > > points > > > > > in > > > > > > a GitHub > > >>>> Discussion first? We can then use that as a > > > reference > > > > > > during the > > >>>> meeting. > > >>>> > > >>>> Regards, > > >>>> > JB > > > > > > > > > > > >>>> > > >>>> On Wed, Sep 9, 2026 at 12:50 AM Dmitri Bourlatchkov > > < > > > > > > > [email protected] > > > > > >>>> wrote: > > >>>> > > > >>>> > Hi > > All, > > > > > > > > > > > > >>>> > > > >>>> > My impression from this thread is that it might > > be > > > > time > > > > > > for a call > > to > > >>>> > discuss all the related issues and > try > > > to > > > > > > arrive at a shared > > >>>> implementation > > >>>> > plan. > > > > >>>> > > > > > > > > > > > > > > > > > > > > > >>>> > I know meetings are not ideal, but in this case it might > be > > > > > > > > > > > > >>>> > > > > > > beneficial as a > > >>>> > means for achieving a common > > understanding > > > > of > > > > > > the set of problems > > and > > >>>> > priorities related to this > > > > > thread. A > > > > > > dedicated metrics meeting > > worked > > >>>> well > > >>>> > > from > > my > > > > > POV. > > > > > > > > >>>> > > > >>>> > Allocating a time slot in the next > community > > > > > meeting > > > > > > might be an > > >>>> option, > > >>>> > although I think we might > > > need > > > > > the > > > > > > full hour in this case. > > >>>> > > > >>>> > WDYT? > > >>>> > > > > > > > > >>>> > > > > > > > Thanks, > > >>>> > Dmitri. > > >>>> > > > >>>> > On Mon, Aug 17, > > 2026 > > > > at > > > > > > 6:39 PM Prithvi S < > > >>>> [email protected]> > > > > >>>> > > > > > > > wrote: > > >>>> > > > >>>> > > Hi all, > > >>>> > > > > >>>> > > > I > > > > > rewrote > > > > > > https://github.com/apache/polaris/pull/5035, > following > > > >>>> > > > the > > > > > > > > > > > > > > > > > >>>> > > discussion to focus only on the SPI foundation. > > >>>> > > > > > > > > > > > > > > > > >>>> > > What is in the PR now: > > >>>> > > > > >>>> > > - A > > written > > > > > > manager-level consistency contract in > > >>>> > > > > > > > > site/content/in-dev/unreleased/persistence-consistency.md. > > > >>>> > > > > > > > > > - > > > > > > CAS-aware EntityMutation / GrantMutation records. > > >>>> > > - > > > > > > MetaStoreChangeSet and BasePersistence#commitChangeSet, > with > > > > > > >>>> > > > > > > backends > > >>>> > > opting in via supportsAtomicMixedCommit(). > > > > > > > > > > >>>> > > > > > > > > - commitChangeSet implementations for the in-memory TreeMap > > > > > >>>> > > > > > > backend and > > >>>> > > JDBC. > > >>>> > > - > > > > > > AtomicOperationMetaStoreManager and > > >>>> > > > > > > TransactionalMetaStoreManagerImpl > > >>>> > > grant/revoke > paths > > > now > > > > > build > > > > > > one change set per operation, > > >>>> falling back > > >>>> > > > to > > > > > > > > > > > >>>> > > individual operations when the backend does not support > > > mixed > > > > > > > > > > > > > > > > > >>>> commits. > > >>>> > > - Tests for the change-set API and > > TreeMap > > > > > > atomic commits. > > >>>> > > > > >>>> > > I intentionally left > out: > > > > > > > > > > > >>>> > > > > >>>> > > - Entity deletes in MetaStoreChangeSet. > > > > > >>>> > > > > > > > > > > > > > > > > > - Refactoring createCatalog, dropEntity, and renameEntity to > > > > use > > > > > > > > > > > > > > > >>>> change > > >>>> > > sets. > > >>>> > > - NoSQL support for > > > > > > commitChangeSet. > > >>>> > > > > >>>> > > I am treating this as > > > Phase > > > > 1 > > > > > so > > > > > > the contract and the SPI > > primitive > > >>>> can be > > >>>> > > > > > > > > > reviewed before the larger operation migrations land. > > >>>> > > > > > > > > > > > > > > > >>>> > > > > > > > > Could you please take a look? In particular, I would like to > > > > > know > > > > > > > > > > > > > >>>> whether > > >>>> > > the contract captures the consensus so > > far, > > > > or > > > > > if > > > > > > it needs to > > >>>> address retry > > >>>> > > / external-work > > > > > > coordination before we merge this foundation. > > >>>> > > > > > >>>> > > > > > > > > > > > Thanks, > > >>>> > > Prithvi S > > >>>> > > > > >>>> > > On Mon, > > Aug > > > > 17, > > > > > > 2026 at 7:40 PM Dmitri Bourlatchkov < > > >>>> [email protected]> > > > > > > > > > >>>> > > > > > > > > wrote: > > >>>> > > > > >>>> > > > Hi Robert, > > >>>> > > > > > > > > > > > > > >>>> > > > > > > > > > I agree that the client-visible operation (e.g. REST API > > > > > > > request) > > > > > > > > >>>> > > consists > > >>>> > > > of many distinct phases. The > > > > database > > > > > / > > > > > > persistence changes > are > > >>>> just one > > >>>> > > of > > > > >>>> > > > > > > > > > > > > > > > > > > > > > these phases. STS (as an example) is another. My point about > > > > >>>> > > > > > > transactions > > >>>> > > > in JDBC Persistence applies to the > > former > > > > > > (database changes) > > >>>> phase - one > > >>>> > > > retry > > attempt > > > > > there > > > > > > should ideally involve exactly one > > >>>> transaction. > > > >>>> > > > > > > > > > > > > > > > > > > > >>>> > > > We certainly need logic inside Polaris Servers to to > > > > > > > coordinate > > >>>> requests > > >>>> > > to > > >>>> > > > > various > > > > > > external systems depending on outcomes from previous > > >>>> > > request > > > > > > > > > > > > >>>> > > > processing phases. > > >>>> > > > > > >>>> > > > I > also > > > > agree > > > > > > that validation based on database state should be > > >>>> > repeated > > > > > > > > > > > >>>> > > > from scratch if we retry the database changes. > > > >>>> > > > > > > > > > > > > > > > > > > > > > > > > > >>>> > > > However, internal RBAC authorization naturally has > to > > > > > happen > > > > > > > > >>>> inside the > > >>>> > > > database update phase > > > > > (read-check-write). > > > > > > IIRC, that is part > of > > >>>> the > > >>>> > > > implicit > > > > consistency > > > > > > guarantees, which were discussed during > > early > > >>>> > > > > > project > > > > > > > > > > > > > > > > > >>>> > > > months (but never got written down in full clarity, > > > > > > >>>> > > > > > > unfortunately). If > > >>>> > > the > > >>>> > > > Authorization > > > check > > > > is > > > > > > inside that phase for internal RBAC, I > > >>>> suppose it > > > > >>>> > > > > > > > > > > > > > will be there for other Authorizers too and that can involve > > > > >>>> > > > > > > external > > >>>> > > > systems (e.g. OPA / Ranger). We can > > certainly > > > > > > expect > > >>>> authorization to be > > >>>> > > > efficient, but > > I'm > > > > not > > > > > > sure how quick it can be in practice. > > >>>> There's > > >>>> > > > > > > > > > > > certainly potential for delays in some (perhaps infrequent) > > > > > cases. > > > > > > > > > > > > > > > > > >>>> > > > > > >>>> > > > Cheers, > > >>>> > > > Dmitri. > > > >>>> > > > > > > > > > > > > > > > > > > > >>>> > > > > > >>>> > > > On Mon, Aug 17, 2026 at 9:20 AM Robert > > > Stupp > > > > < > > > > > > [email protected]> > > >>>> wrote: > > >>>> > > > > > >>>> > > > > > Hi > > > > > > Dmitri, > > >>>> > > > > > > >>>> > > > > I agree that one > logical > > > > > > operation needs a consistent view > of > > >>>> the whole > > > >>>> > > > > > > > > > > > > > > > > > > > > > > > > backend state. > > >>>> > > > > What worries me is treating one > > long > > > > JDBC > > > > > > transaction as the > > >>>> boundary > > >>>> > > of > > >>>> > > > > > > > > > > > that > > > > > > operation and then retrying the complete request if the > > >>>> > > > commit > > > > > > > > > > > > > >>>> > > > fails. > > >>>> > > > > > > >>>> > > > > The request > can > > > > also > > > > > > write metadata to object storage or > call > > >>>> STS. > > >>>> > > > > > > > > > > > > > > > > Each system can leave us with an outcome whose certainty we > > > > >>>> > > > > > cannot > > > > > > > > >>>> > > know. > > >>>> > > > > The database may have > committed > > > > > before > > > > > > the connection was > > lost, > > >>>> an > > >>>> > > > > > > > object-store > > > > > > write may have succeeded before a timeout, or > STS > > >>>> may > > have > > > > > > > > > > > > >>>> > > > > issued credentials before its response was lost. > > > > > >>>> > > > > > > > > > > > > > > > > > > > A database rollback cannot undo any of those effects, and > > > > > >>>> > > > > > > retrying the > > >>>> > > > > whole request may repeat them. > > > > > >>>> > > > > > > > > > > > > > > > > > >>>> > > > > So I think the contract has to separate the > > > overall > > > > > > > > > > > operation > > >>>> from each > > >>>> > > > > attempt to commit > it. > > > > > > > > > > > >>>> > > > > Each database or backend commit attempt should stay > > > short. > > > > > > > > > > > > > >>>> > > > > When we retry, validation and authorization need to > > use > > > > the > > > > > > > > > > > > > >>>> state for > > >>>> > > > that > > >>>> > > > > new > attempt, > > > and > > > > we > > > > > > need clear rules for which external work > > >>>> can be > > > >>>> > > > > > > > > > > > > reused, > > >>>> > > > > repeated, cleaned up, or reconciled > after > > an > > > > > > uncertain > > outcome. > > >>>> > > > > > > >>>> > > > > Some > > > > contextual > > > > > > state could be useful for carrying that > state. > > >>>> > > > > > > But > > > > the > > > > > > JDBC connection and transaction should be an > > >>>> > > implementation > > > > > > > > > > > > > > > >>>> > > > detail > > >>>> > > > > inside it, not the boundary of > > the > > > > > > complete operation. > > >>>> > > > > > > >>>> > > > > Cheers, > > > > > >>>> > > > > > > > > > > > > > > > > > > > Robert > > >>>> > > > > > > >>>> > > > > > > >>>> > > > > On > > Mon, > > > > Aug > > > > > > 10, 2026 at 11:04 PM Dmitri Bourlatchkov < > > >>>> > > [email protected] > > > > > > > > > > > > >>>> > > > > > >>>> > > > > wrote: > > >>>> > > > > > > >>>> > > > > > > > > > > > > > > > Hi > > > > > > All, > > >>>> > > > > > > > >>>> > > > > > The idea of a > > > > backend-agnostic > > > > > > change-set primitive sounds > > >>>> good. > > >>>> > > > > > > > > > > >>>> > > > > > > > > > > > > > > > > > > > > However, I am not sure it is sufficient for all the use > > > > > cases > > > > > > > > > > > > > >>>> we > > >>>> > > > touched > > >>>> > > > > > here. More > > > > > specifically, > > > > > > the state read by validation code > > is > > >>>> not > > >>>> > > > > > > > > > > > > > > > > > > > > > necessarily reflected in the change set: > > >>>> > > > > > > > > > >>>> > > > > > > > > > > > > > > > > > > > > > > > * Unchanged entities might be considered by validation, > but > > > > > > > > > >>>> > > > > > > changed > > >>>> > > in > > >>>> > > > a > > >>>> > > > > > > > parallel > > > > > > request. > > >>>> > > > > > * Even for changed entities the state > > > read > > > > by > > > > > > validation > may > > >>>> differ > > >>>> > > > from > > >>>> > > > > > > > > > > > > > > the > > > > > > state being altered. Some validation code talks to the > > >>>> > > > > > MetaStore > > > > > > > > >>>> > > > > > directly, outside of the data produced by the > > > > Resolver. > > > > > > > > > > > > > >>>> > > > > > > > >>>> > > > > > I do not think Polaris offers > > any > > > > > > explicit mechanisms > (ATM) > > >>>> to ensure > > >>>> > > > > > > > > > > > consistency between these reads and subsequent writes. > > >>>> > > > > > > > > > > > > > > > > > > > > > > >>>> > > > > > As far as JDBC goes, running an overarching > > > > Transaction > > > > > > > > > > > > > >>>> across all > > >>>> > > > reads > > >>>> > > > > > and > > writes > > > in > > > > > the > > > > > > same request, at the SERIALIZABLE > > isolation > > >>>> level > > > > > > >>>> > > > > > > > > > > > > > in > > >>>> > > > > the > > >>>> > > > > > backing RDBMS could > > solve > > > > the > > > > > > problem, I think. > > >>>> > > > > > > > >>>> > > > > > Indeed, > > such > > > a > > > > > > transaction might be long to accommodate > > calls > > >>>> made > by > > > > > > > > > > > >>>> > > > > > Polaris to external storage, etc. However, is > that a > > > > > > > > > > > problem? > > >>>> I think > > >>>> > > > the > > >>>> > > > > > > > > > > alternative > > > > > > is for the JDBC Persistence impl. to > "manually" > > >>>> track > > all > > > > > > > > > > > > > > > >>>> > > > > reads > > >>>> > > > > > under the same request and > > redo > > > > > them > > > > > > in the small > > transaction > > >>>> that > > >>>> > > > > > > persists > > > > > > > > > > > > > > > >>>> > > > > > the writes. This will also add RDBMS overhead and > > > > require > > > > > > > > > > > > > >>>> complex > > >>>> > > code > > >>>> > > > in > > >>>> > > > > > > > > > > > > > Polaris to handle the data properly. > > >>>> > > > > > > > >>>> > > > > > > > > > > > > > > > > > > > > > > > > > > Tracking the request-wide transaction can be done only for > > > >>>> > > > > JDBC > > > > > > > > > > > > > >>>> > > without > > >>>> > > > > > leaking "transaction" > > concepts > > > to > > > > > the > > > > > > NoSQL Persistence, I > > >>>> think. > > >>>> > > NoSQL > > >>>> > > > > > > > > > > > > > > > > > > > > > > > > > > will use other mechanisms to ensure read/write > consistency. > > > > > >>>> > > > > > > > > > > > > > > > > > > > > > > >>>> > > > > > Connecting to Robert's email (a parallel > > > > branch > > > > > in > > > > > > this > > >>>> discussion), > > >>>> > > > I'd > > >>>> > > > > > > > like > > > > to > > > > > > propose this approach: > > >>>> > > > > > > > >>>> > > > > > * > Each > > > > > request > > > > > > establishes a "Data Context" > > >>>> > > > > > - In JDBC the > Data > > > > > Context > > > > > > corresponds to a JDBC > > Connection > > >>>> + Tx > > >>>> > > > > > > > > > - > > > > > In > > > > > > NoSQL the Data Context tracks one or more reference > > >>>> > hashes > > > > > > > > > > > >>>> > > > > > * All Persistence access in the same request goes > > > > through > > > > > > > > > > > > > the > > >>>> same > > >>>> > > Data > > >>>> > > > > > Context > > > > > > > > > > >>>> > > > > > > > > > > > * All Persistence changes are committed once at the end > of > > > > > > > > > > > > the > > > > > > > > >>>> > > request > > >>>> > > > > > - Not all changes have to > > be > > > > > > globally atomic. For > example, > > >>>> changes > > >>>> > > in > > > > > > > > > > >>>> > > > > > > > > > > > Catalogs A and B do not have to be atomic with respect to > > > > > > > > > >>>> > > > > > > each other. > > >>>> > > > We > > >>>> > > > > > can go deeper > into > > > > this > > > > > > later. This is relevant to NoSQL. > > >>>> > > > > > - > > Transactional > > > > > > backends like JDBC can, of course, > choose > > >>>> to make > > > > >>>> > > > > > > > > > > > > all > > >>>> > > > > > changes globally atomic. > > >>>> > > > > > > > > * > > > A > > > > > > commit can fail in two main ways: > > >>>> > > > > > - A > retriable > > > > > failure > > > > > > like an optimistic lock error or > Tx > > >>>> > > > > > > > serializability > > > > > > > > > > > > > > > > > >>>> > > > > > error > > >>>> > > > > > - A non-triable logical > > error > > > > > (e.g. > > > > > > entity not found) > > >>>> > > > > > * On a retriable error the > > whole > > > > > > request is re-attempted > (as > > >>>> if > > >>>> > > > > > > > resubmitted > > > > > > > > > > > > > > > > > >>>> > > > > > by a client) a few times (configurable timeout). > > > > > > > >>>> > > > > > > > > > > > > > > > > > > >>>> > > > > > WDYT? > > >>>> > > > > > > > >>>> > > > > > > > > > > > > > > > > > Cheers, > > >>>> > > > > > Dmitri. > > >>>> > > > > > > > >>>> > > > > > > > > > > > > > > > > > > > > On > > > > > > Sun, Jul 26, 2026 at 12:44 PM Jean-Baptiste Onofré < > > >>>> > > > > > > > > [email protected] > > >>>> > > > > > > >>>> > > > > > wrote: > > > > >>>> > > > > > > > > > > > > > > > > > > > > > > > > > >>>> > > > > > > Hi all > > >>>> > > > > > > > > >>>> > > > > > > > > > > > > > > > I > > > > > > think Robert has a good point. > > >>>> > > > > > > > > >>>> > > > > > > > > > > > > > > > > > > > > > If > > > > > > the atomicity guarantee lives only in > BasePersistence, > > >>>> > > > then > > > > > the > > > > > > > > >>>> > > > > > > manager contract can't tell a caller whether > > the > > > > > state > > > > > > > it > > >>>> read for > > >>>> > > > > > > > > > > > > validation/authorization/credential-vending is the same > > >>>> > > > state > > > > > that > > > > > > > > >>>> > > > > > > eventually commits. That's the actual bug > > class > > > > > behind > > > > > > > the > > >>>> JDBC > > >>>> > > > > > > symptoms (not any single > > > > > operation > > > > > > being non-atomic, but > > the > > >>>> > > contract > > >>>> > > > > > > > > > > > > > > > > > > > > > > being silent about it. Every operation-specific fix > (like > > > > >>>> > > > > > #4939, > > > > > > > > >>>> > > > #5035 > > >>>> > > > > > > or #5095) re-answers > this > > > > > > question locally and it keeps > > >>>> recurring. > > >>>> > > So > > > > > > > > > >>>> > > > > > > > > > > > > the deliverable should start with a written > consistency > > > > > > > > > > > >>>> contract at > > >>>> > > > > > > the manager level. > > > >>>> > > > > > > > > > > > > > > > > > > > > > > >>>> > > > > > > I'm not sure migrating everyone to > > > >>>> > > > > > > > > > > > > > TransactionalMetaStoreManagerImpl > > >>>> > > > > > > is a good > > > idea. > > > > It > > > > > > would tie the logical change set to a > > DB > > >>>> > > > > > > > > > > > > transaction spanning the REST request. It means: > > >>>> > > > > > > > > > > - > > > > > it > > > > > > holds a durable transaction open across slow > external > > >>>> > > work > > > > > > > > > > > > >>>> > > > > > > (credential vending, OPA, Ranger, ...) > > >>>> > > > > > > > > > > > > > > > > > > > > > > > > > > > - it doesn't map to NoSQL > > >>>> > > > > > > - It wraps > > > single-row > > > > > > updates in runWiithinTransaction, > > >>>> which is a > > >>>> > > > > > > > > > > > > > > > > > > > > > overhead > > >>>> > > > > > > > > >>>> > > > > > > So, I think > the > > > > > > transactional manager isn't a portable > > >>>> target. It's > > > > > >>>> > > > > > > > > > > > > > > > > > "only" one backend's strategy. > > >>>> > > > > > > > > > >>>> > > > > > > > > > > > > > > > > > > > > > > > > > > I think Privthi's approach is right. A backend-agnostic > > > > >>>> > > > > > > change-set > > >>>> > > > > > > primitive with a documented > > fallback > > > is > > > > > the > > > > > > correct > shape. > > >>>> There is > > >>>> > > > > > > one > caveat: > > > > > without > > > > > > carrying the original entitiy (not > > just > > >>>> the new > > > > > >>>> > > > > > > > > > > > > > > > > > one) the change set can't express optimistic > concurrency. > > > > > > > > > >>>> > > > > > > > > > > > > > > >>>> > > > > > > I propose the following > > multi-steps > > > > > > approach: > > >>>> > > > > > > 1. We write the manager-level > > > > consistency > > > > > > contract as > > >>>> Robert asked. > > >>>> > > > It > > >>>> > > > > > > > > > > > > > > > > > > should include the explicit statement that a logical > > change > > > > > > > >>>> > > > > > > set is > > >>>> > > > not > > >>>> > > > > > > a request-scope DB > > > > > > transaction. > > >>>> > > > > > > 2. We make Compare And Swap > > > > > > (optimistic-concurrency > > pattern) > > >>>> > > baseline > > > >>>> > > > > > > > > > > > > > > > > > > > > first-class in EntityMutation before merging the SPI > > > >>>> > > > > > > > > > > > > > > > > > > > > 3. Then we refactor > createCatalog/dropEntity/renameEntity. > > > > > > >>>> > > > > > > > > > > > > > > > > > > > >>>> > > > > > > Thoughts? > > >>>> > > > > > > > > > > >>>> > > > > > > > > > > > > > > > > > > > > > > > > Regards > > >>>> > > > > > > JB > > >>>> > > > > > > > > > >>>> > > > > > > > > > > > > > > > > > > > > On Fri, Jul 24, 2026 at 5:08 PM Robert Stupp < > > >>>> > > > > > [email protected]> > > > > > > > > >>>> > > wrote: > > >>>> > > > > > > > > > >>>> > > > > > > > > > Hi > > > > > all, > > > > > > > > >>>> > > > > > > > > > >>>> > > > > > > > Yufei's clarification > > > about > > > > > > where the atomicity > > guarantee > > >>>> is > > >>>> > > > > > defined > > > > > > > > > > > > > > > >>>> > > > > > > seems > > >>>> > > > > > > > important. > > > >>>> > > > > > > > > > > > > > > > > > > > > > If it is a property of the lower-level BasePersistence > > > >>>> > > > > > > contract > > >>>> > > > > rather > > >>>> > > > > > > than > > > >>>> > > > > > > > > > > > > > > > > > > > > > the general PolarisMetaStoreManager contract, the > > > general > > > > > > > > > > > >>>> > > contract > > >>>> > > > > does > > >>>> > > > > > > not > > > > > > > > > > >>>> > > > > > > > > > > > > > tell callers whether the state used for validation, > > > > > > > >>>> > > > > > > > > > > > > > authorization, > > >>>> > > > or > > >>>> > > > > > > > > > credential > > > > > > vending is consistent with the change that > > >>>> eventually > > > > > > >>>> > > > > > > > > > > > > > > > > commits. > > >>>> > > > > > > > > > >>>> > > > > > > > The > > > > current > > > > > > PRs suggest that operation-specific > > >>>> multi-object > > > >>>> > > > > > > > > > > > > > > > > > > methods > > >>>> > > > > > can > > >>>> > > > > > > > fix > > individual > > > > > cases, > > > > > > while leaving the same contract > > >>>> question to > > >>>> > > > > > > > > > > > recur > > > > > > > > >>>> > > > > > > for > > >>>> > > > > > > > each new case. > > > > > > >>>> > > > > > > > > > > > > > > > > > > > > >>>> > > > > > > > I would also avoid defining a > > > logical > > > > > > change set as a > > >>>> database > > >>>> > > > > > > > transaction > > > > > > > > > > > > >>>> > > > > > > > > > > > > > around an entire REST request. > > >>>> > > > > > > > > > > > > That > > > > > > would tie the contract to the backend and > database, > > >>>> > and > > > > > > > > > > > > >>>> > > > > > > > > could > > >>>> > > > > keep > > >>>> > > > > > > the > > > >>>> > > > > > > > > > > > > > > > > > > > > > durable attempt open across slow external work. > > >>>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>>> > > > > > > > So is the choice really between the two > > > current > > > > > > > > > > > manager > > >>>> > > > > > implementations, > > >>>> > > > > > > > or > > > > > > > > > > > >>>> > > > > > > > do we first need to revisit the boundary and > > > > > > > guarantees > > >>>> exposed > > >>>> > > to > > >>>> > > > > > > > their > > > > > > > > > > > > > > > >>>> > > > > > > > callers? > > >>>> > > > > > > > > > >>>> > > > > > > > > > > > > > > > > > > > Cheers, > > >>>> > > > > > > > Robert > > >>>> > > > > > > > > > > > > >>>> > > > > > > > > > > > > > > > > > > > > >>>> > > > > > > > On Fri, Jul 17, 2026 at 10:21 PM > > > Dmitri > > > > > > Bourlatchkov < > > >>>> > > > > [email protected] > > >>>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>>> > > > > > > > wrote: > > >>>> > > > > > > > > > >>>> > > > > > > > > > > > > > > > > > > > > Hi Yufei, > > >>>> > > > > > > > > > > >>>> > > > > > > > > > Thanks > > > for > > > > > the > > > > > > link. I stand corrected. The "one > > atomic > > >>>> change > > > > > >>>> > > > > > > > > > > > > > > per > > >>>> > > > > > > method" > > >>>> > > > > > > > > > contract > > as > > > > > > defined in javadoc does apply only to > > >>>> > > > BasePersistence > > > > > > > > > > > > >>>> > > > > > > > > > > > > > > and AtomicOperationMetaStoreManager (which > > > delegates > > > > > > > > > > > to > > > > > > > > >>>> > > > > > > BasePersistence). > > >>>> > > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > > > > > > > > Note that all non-test Persistence implementations > > > > in > > > > > > > > > > > > >>>> the > > >>>> > > Polaris > > >>>> > > > > > > codebase > > > > >>>> > > > > > > > > > > > > > > > > > > > > > > > > > > extend those classes (which is probably why I was > > > > >>>> > > > > > > confused > > >>>> > > about > > >>>> > > > > > > atomicity > > > >>>> > > > > > > > > > > > > > > > > > > > > > > expectations). > > >>>> > > > > > > > > > > >>>> > > > > > > > > > > > > > > > > > > > > > > > > However, this creates a gap in the Persistence SPI > > >>>> > > > > > > > > specification. > > >>>> > > > > If > > >>>> > > > > > > > > other > > > > > > PolarisMetaStoreManager implementations do not > > >>>> have to > > > > > > > >>>> > > > > > > > > > > > > > > comply > > >>>> > > > > > > with > > >>>> > > > > > > > > > this > > > > > > principle, it will create a conceptual > difficulty > > >>>> at > > call > > > > > > > > > > > > > > > >>>> > > > > sites. > > >>>> > > > > > > How > > >>>> > > > > > > > > > > > > > > can > > > > > > PolarisMetaStoreManager callers reason about > > >>>> consistency > > > > > > > > > >>>> > > > > > > > > and > > >>>> > > > > > > durability > > >>>> > > > > > > > > > > > > > behaviours > > > > > > in general? > > >>>> > > > > > > > > > > >>>> > > > > > > > > I > > > believe > > > > > we > > > > > > need to address that as part of this > > >>>> discussion. > > > >>>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>>> > > > > > > > > Cheers, > > >>>> > > > > > > > > > > > > > Dmitri. > > > > > > > > >>>> > > > > > > > > > > >>>> > > > > > > > > On Thu, Jul 16, > > 2026 > > > at > > > > > > 9:29 PM Yufei Gu < > > >>>> [email protected] > > >>>> > > > > > > > > > > > > >>>> > > > > > > > > > > > wrote: > > >>>> > > > > > > > > > > >>>> > > > > > > > > > > > > > > > > > > > > > The > > > > > > MetaStore SPI is currently defined with the > > >>>> idea that > > > > > > >>>> > > > > > > > > > > > > > one > > >>>> > > > > > > method > > >>>> > > > > > > > > call > > > > > >>>> > > > > > > > > > > > > > > > > > > > > > > > > means one atomic change. > > >>>> > > > > > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > > > > If the "MetaStore SPI" refers to the interface > > > > > > > > > > >>>> > > > > > > > > > > > > PolarisMetaStoreManager, I > > >>>> > > > > > > > > > > > don't > > > > > think > > > > > > we've ever state each method to be > > >>>> atomic. We did > > > >>>> > > > > > > > > > > > > > > > > > > > clarify > > >>>> > > > > > > > > > atomicity[1] in the > interface > > > > > > BasePersistence > > though. > > >>>> > > > > > > > > > > > >>>> > > > > > > > > > > > > > > > > > > > > > > > > > > >>>> > > > > > > > > > 1. > > >>>> > > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > > > > > > > > > > > >>>> > > > > > > > > > > >>>> > > > > > > > > > > >>>> > > > > > > > > > > > > > > > > > > > > > > > > > >>>> > > > > > > >>>> > > > > > >>>> > > > > >>>> > > > > > > > > > > > > > > > > > > https://github.com/apache/polaris/blob/e9039e12003a13e783b51 > > > 30a3d30d30cfe78d93c/polaris-core/src/main/java/org/apache/ > > > polaris/core/persistence/BasePersistence.java#L48 > > > > > > > > >>>> > > > > > > > > > > > >>>> > > > > > > > > > > > >>>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>>> > > > > > > > > > Yufei > > >>>> > > > > > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > > > > > > >>>> > > > > > > > > > On Thu, Jul 16, 2026 > > at > > > > > > 11:27 AM Dmitri > > Bourlatchkov < > > >>>> > > > > > > > > > > [email protected] > > > > > > > > > > > > > > >>>> > > > > > > > > > wrote: > > >>>> > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > > > > > > > > > > Hi Yufei, > > >>>> > > > > > > > > > > > > >>>> > > > > > > > > > > > > > > > > > > > > > > > > > I agree that JDBC transactions must be handled > > more > > > > > >>>> > > > > > > > > > > > > > > > > > > explicitly. > > >>>> > > > > > > > > However, > > >>>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I'm not sure that simply moving to > > >>>> > > > > > > > > > > > > TransactionalMetaStoreManagerImpl is > > >>>> > > > > > > > > > > > > > > > > > sufficient. > > >>>> > > > > > > > > > > > > >>>> > > > > > > > > > > > > > > > > The > > > > > > MetaStore SPI is currently defined with the > > >>>> idea that > > > > > > >>>> > > > > > > > > > > > > > one > > >>>> > > > > > > method > > >>>> > > > > > > > > call > > > > > >>>> > > > > > > > > > > > > > > > > > > > > > > > > > means one atomic change [1]. The "transactional" > > > > > > > > > >>>> > > > > > > MetaStore > > >>>> > > > > impl. > > >>>> > > > > > is > > >>>> > > > > > > > > > > > > > > > > > > > > > > > but a > > >>>> > > > > > > > > > > sub-case of that. It > cannot > > > > alter > > > > > > the high-level > > >>>> contract. > > >>>> > > > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > > > > > > > > > > We could add SPI methods having multiple object > > > > > > > >>>> > > > > > > parameters > > >>>> > > to > > >>>> > > > > > > represent > > > >>>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > grouped changes, but I am not sure it will be a > > > > >>>> > > > > > sound > > > > > > > > >>>> > > design. > > >>>> > > > > > This > > >>>> > > > > > > > > > will > > > > > > > > > > > > > > > > > >>>> > > > > > > > > > > bloat the interface surfaces and require > > > extra > > > > > > > > > > > > > >>>> impl. effort > > >>>> > > > for > > >>>> > > > > > > each > > > > > > >>>> > > > > > > > > > > > > > > > > > > > > > backend type. More importantly, adding multi-arg > > > > > > > > > >>>> > > > > > > change > > >>>> > > > methods > > >>>> > > > > > > still > > > >>>> > > > > > > > > > > > > > > > > > > > > > > > won't > > >>>> > > > > > > > > > > address the problem of > > > reads > > > > > > being consistent > with > > >>>> writes, > > >>>> > > > > > > because > > > > > > > > > > > >>>> > > > > > > each > > >>>> > > > > > > > > > > method call > will > > > > still > > > > > > be independent regarding > > the > > >>>> data > > >>>> > > > > > stored > > > > > > > > > > > > > > > >>>> > > > > > in > > >>>> > > > > > > the > > >>>> > > > > > > > > > > > > > > > > > > > > database. > > >>>> > > > > > > > > > > > > >>>> > > > > > > > > > > > > I > > > > > tend > > > > > > to think we need to introduce a "change > > set" > > >>>> or > > > > > >>>> > > > > > > > > > > > > > > "atomic > > >>>> > > > > > > batch" > > >>>> > > > > > > > > > > > > > > concept > > > > > to > > > > > > core Persistence and associate each > > REST > > >>>> API > > > >>>> > > > > > > > > > > > > > > > > > > request > > >>>> > > > > > > with > > >>>> > > > > > > > > one > > > > > > >>>> > > > > > > > > > > > > > > > > > > > > > such change set, which will be committed (or > > > > > rolled > > > > > > > > > > > > > >>>> back) > > >>>> > > at > > >>>> > > > > the > > >>>> > > > > > > > > > > > > > end > > > > > of > > > > > > > > >>>> > > > > > > > > > the > > >>>> > > > > > > > > > > > > request. > > > I > > > > > > believe Ayush mentioned a similar > > >>>> concept in PR > > > >>>> > > > > > > > > > > > > 4939 > > >>>> > > > > > > [2]. In > > >>>> > > > > > > > > > > > JDBC > > > > each > > > > > > change set will naturally be > associated > > >>>> with an > > > > >>>> > > > > > > > > > > > > > RDBMS > > >>>> > > > > > > > > > > transaction. In NoSQL > > persistence, > > > > > each > > > > > > atomic > > >>>> change set > > >>>> > > > will > > >>>> > > > > > be > > > > > > > > > > > > >>>> > > > > > > > > > > > > > > > > associated with one CAS operation on the > > > > > > > underlying > > > > > > > > >>>> > > database. > > >>>> > > > > > > > > > > > > >>>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > [1] > > >>>> > > > > > > > > >>>> > > > > > > https://lists.apache.org/thread/rf5orxs815zs4h64p4rwp03q3pbgxb5r > > > > > > > > > >>>> > > > > > > > > > > > > > > > > > > >>>> > > > > > > > > > > [2] > > >>>> > > > > > > > > > > > > > > > > > > > > > > > >>>> > > > > > https://github.com/apache/polaris/pull/4939#discussion_r3575719158 > > > > > > > > >>>> > > > > > > > > > > > > >>>> > > > > > > > > > > Cheers, > > > > > > > > > >>>> > > > > > > > > > > > > > > > > Dmitri. > > >>>> > > > > > > > > > > > > > >>>> > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Jul 16, 2026 at 1:12 PM Yufei Gu < > > >>>> > > > > > > > > > > [email protected] > > >>>> > > > > > > > >>>> > > > > > > > > wrote: > > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > >>>> > > > > > > > > > > > Thanks > for > > > > > raising > > > > > > this, Dmitri. > > >>>> > > > > > > > > > > > > > >>>> > > > > > > > > > > > > > > > > > > > > > > > > > > These are valid concerns, and they were > already > > >>>> > > > > > > > recognized > > > > > > > > >>>> > > > > when > > >>>> > > > > > > we > > >>>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > introduced JDBC persistence to Polaris. At > that > > >>>> time, > we > > > > > > > > > > > >>>> > > > > chose > > >>>> > > > > > > to use > > >>>> > > > > > > > > > > > > > > > > > > > > > > > > > > AtomicOperationMetaStoreManager for the JDBC > due > > >>>> to > > > the > > > > > > > > > > > > >>>> > > > > > > simplicity. I > > >>>> > > > > > > > > > > think > > > > > > > > > >>>> > > > > > > > > > > > > > > > > > most of the issues mentioned here can > > already > > > > > be > > > > > > > > > > > > > >>>> > > addressed > > >>>> > > > by > > >>>> > > > > > > > > > > > > > > > > > > > TransactionalMetaStoreManagerImpl. > > >>>> > > > > > > > > > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > > > > > > For example, rename is already wrapped in a > > > > > > > > > >>>> > > > > > > transaction > > >>>> > > in > > >>>> > > > > > > > > > > > > > > > > > TransactionalMetaStoreManagerImpl [1]. > > Similarly, > > >>>> > > > catalog > > > > > > > > > > > > > > > > > >>>> > > > > > > creation, > > >>>> > > > > > > > > > which > > > > >>>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > involves reading and creating multiple > objects, > > > > > > > > > >>>> > > > > > > is also > > >>>> > > > > > executed > > >>>> > > > > > > > > > > > > within a > > > > > > > > > > > > > >>>> > > > > > > > > > > > transaction [2]. > > >>>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>>> > > > > > > > > > > > I see two possible directions: > > > > > >>>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>>> > > > > > > > > > > > 1. > > >>>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>>> > > > > > > > > > > > Modify > > > > > > AtomicOperationMetaStoreManager > > >>>> together with > > >>>> > > > > > > the > > > > > > > > > > > > > >>>> > > > > > > > > persistence > > >>>> > > > > > > > > > > > > > > > backends > > > > > > (such as JDBC) to provide the > > required > > >>>> > > > > > > > consistency > > > > > > > > > > >
