Hi all,
Some updates on this:
I recently started a separate discussion regarding dynamic datasource
activation at runtime [1]. This was intended as a prerequisite for
adopting H2 as the default JDBC driver, as suggested on this thread.
However, that proposal may end up being rejected.
Given this situation, we should reconsider our options. If the
PostgreSQL + H2 approach is no longer a viable replacement to the
TreeMapMetaStore, I propose an alternative: let's use the NoSQL
metastore with an InMemory backend.
My reasoning is as follows:
1. It remains a production-ready metastore, so strictly superior to
TreeMapMetaStore.
2. The metastore state update logic is the same regardless of the
backend, so we're still close to a real production setup (same as with
an in-memory JDBC driver).
3. It doesn't require any external dependencies (no bundled driver).
I'd also note that while the NoSQL metastore is already included
today, it is not actionable because no backend is available. This goes
against the onboarding UX that we're trying to achieve, as users
interested in Polaris with the NoSQL metastore cannot test this setup
with the official image.
Therefore, I think NoSQL + InMemory would serve as a more appropriate
default setup for the official Polaris image.
>From what I see, it would boil down to adding one line to
polaris-server or polaris-service:
runtimeOnly(project(":polaris-persistence-nosql-db-inmemory"))
What are your thoughts on this?
Thanks,
Alex
[1]: https://lists.apache.org/thread/jy6wb186h94n9q86kv01shbn68ppr6gv
On Fri, Jun 19, 2026 at 12:07 PM Alexandre Dutra <[email protected]> wrote:
>
> Hi all,
>
> I wanted to give an update on this topic: I have started working on
> this, but I would like to have [4812] merged first, since that's a
> prerequisite.
>
> Thanks,
> Alex
>
> [4812]: https://github.com/apache/polaris/pull/4812
>
> On Tue, Jun 16, 2026 at 5:15 PM Russell Spitzer
> <[email protected]> wrote:
> >
> > I believe the original idea of TreeMapMetastore was that it looked very
> > similar to FoundationDB from an API perspective, so it served well as a
> > test system for the original backend being developed at SF.
> >
> > I agree that H2 + JDBC makes sense to me now for the project.
> >
> > On Tue, Jun 16, 2026 at 10:03 AM Alexandre Dutra <[email protected]> wrote:
> >
> > > Hi all,
> > >
> > > Thank you for your feedback.
> > >
> > > There seems to be a general agreement on the idea of deprecating
> > > TreeMapMetaStore, coupled with the JDBC + H2 solution for tests.
> > >
> > > In most modules, the tests migration won't pose any serious
> > > challenges, as replacing the metastore is generally just a matter of
> > > changing the configuration, and making sure the realm is properly
> > > bootstrapped.
> > >
> > > There will be, however, a few tricky situations in polaris-core: a few
> > > tests rely on the TreeMapMetaStore, mostly as a test convenience; but
> > > there is no obvious replacement for it in that module. I am however
> > > confident that we can find a solution for that, either based on mocks,
> > > or by bringing in a real metastore.
> > >
> > > If no objections are raised, I am going to prepare a PR for this.
> > >
> > > Thanks,
> > > Alex
> > >
> > > On Tue, Jun 16, 2026 at 3:28 PM Dmitri Bourlatchkov <[email protected]>
> > > wrote:
> > > >
> > > > Hi Alex,
> > > >
> > > > Thanks for starting this thread.
> > > >
> > > > I also find the TransactionalMetaStoreManagerImpl and related call paths
> > > > conuising in Apache Polaris code.
> > > >
> > > > I support promoting H2 to the default (in memory) Persistence backend
> > > > for
> > > > getting started cases. This should also resolve the old H2 evolution
> > > thread
> > > > [1] by ensuring it is used regularly on the same code paths as
> > > PostgreSQL.
> > > >
> > > > Deprecating TransactionalMetaStoreManagerImpl for removal also sounds
> > > > reasonable to me. Existing downstream users
> > > > of TransactionalMetaStoreManagerImpl will have time to migrate, or even
> > > > embed that code (per ASF license) into local builds, during the
> > > deprecation
> > > > phase.
> > > >
> > > > [1] https://lists.apache.org/thread/g1gg2w8hn9gvlmwrdh0x218whoh2wd39
> > > >
> > > > Cheers,
> > > > Dmitri.
> > > >
> > > > On Fri, Jun 12, 2026 at 11:24 AM Alexandre Dutra <[email protected]>
> > > wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > I am writing to ask the community whether it is OK to deprecate
> > > > > TreeMapMetaStore, as well as all the in-memory metastore manager,
> > > > > metastore manager factory, and persistence types that rely solely on
> > > > > TreeMapMetaStore.
> > > > >
> > > > > As we know, these components are test-grade only, and not suitable for
> > > > > production. They trigger a production readiness alert on Polaris
> > > > > startup. It's a considerable amount of code that is virtually dead in
> > > > > production.
> > > > >
> > > > > It's also confusing for developers. E.g. the "transactional" metastore
> > > > > is not transactional in the JDBC sense of the term, and thus not used
> > > > > by JDBC persistence. It also has its quirks: some return statuses are
> > > > > only returned by that manager.
> > > > >
> > > > > However, these components are used in tests, and I agree that it's
> > > > > useful to have an in-memory version of the persistence layer for
> > > > > tests. But we have today two alternatives that are imho superior for
> > > > > tests in polaris-runtime-service:
> > > > >
> > > > > - JDBC persistence with H2 backend. There are already a few tests
> > > > > using this setup.
> > > > > - NoSQL persistence with InMemory backend.
> > > > >
> > > > > Both alternatives test real production-grade persistence code.
> > > > >
> > > > > And finally, TreeMapMetaStore is currently the default runtime
> > > > > persistence in application.properties; and the Helm chart also
> > > > > advertises it as the default. These are not sane defaults, imho. It's
> > > > > always tricky to provide a good default for datastores, but since JDBC
> > > > > persistence is included by default in the server image, I think that
> > > > > including the H2 driver by default could give us a saner default while
> > > > > keeping the out-of-the-box experience intact (the license is Category
> > > > > A).
> > > > >
> > > > > Concretely:
> > > > >
> > > > > On the MetaStoreManagerFactory hierarchy, the following
> > > > > implementations are completely in-memory:
> > > > >
> > > > > - InMemoryPolarisMetaStoreManagerFactory: could be removed
> > > > >
> > > > > - InMemoryAtomicOperationMetaStoreManagerFactory: could be removed
> > > > >
> > > > > - LocalPolarisMetaStoreManagerFactory: is the base class of the two
> > > > > above; imho it can be removed, but since it's an abstract class, it
> > > > > may have been extended outside Polaris. But neither JDBC nor NoSQL use
> > > > > it.
> > > > >
> > > > > On the PolarisMetaStoreManager hierarchy:
> > > > >
> > > > > - TransactionalMetaStoreManagerImpl: is only used by
> > > > > LocalPolarisMetaStoreManagerFactory. JDBC and NoSQL do not use it.
> > > > > Could be removed.
> > > > >
> > > > > - TransactionWorkspaceMetaStoreManager however is a different beast,
> > > > > in spite of the similar name. It is in use today on the commit path,
> > > > > so should not be removed.
> > > > >
> > > > > On the BasePersistence hierarchy:
> > > > >
> > > > > - TreeMapTransactionalPersistenceImpl and its TreeMapMetaStore are
> > > > > only used by InMemoryPolarisMetaStoreManagerFactory, and could be
> > > > > removed;
> > > > >
> > > > > - TransactionalPersistence and AbstractTransactionalPersistence: these
> > > > > are supertypes of TreeMapTransactionalPersistenceImpl and thus only
> > > > > used for in-memory. They imo can be removed, but they might have been
> > > > > extended or implemented outside Polaris.
> > > > >
> > > > > I think it's important to keep Polaris code tidy by removing unused,
> > > > > unimplementable, or test-grade only components.
> > > > >
> > > > > What are your thoughts?
> > > > >
> > > > > Thanks,
> > > > > Alex
> > > > >
> > >
