Thanks, Prashant, for working on this! Using JSONB in the WIP JDBC implementation makes sense. For the EclipseLink implementation, I'd recommend keeping the current schema as-is (without JSONB) so existing users don’t need to go through a migration.
If users do want to migrate, I’d suggest moving directly to the JDBC implementation rather than adapting EclipseLink to the new JSONB field Yufei On Fri, Mar 21, 2025 at 9:08 AM Prashant Singh <prashant.si...@snowflake.com.invalid> wrote: > Hi Robert, > > Thank you for your response! > > To clarify my point about "non-transactional" operations, I meant that most > operations don't require full transactions, with the exception of methods > such as writeEntities. For this specific case, Dennis created a new > implementation of Polaris MetaStore Manager, AtomicMetaStoreManager [1], > which: > > - Implements PolarisMetaStoreManager using only one-shot atomic > operations within a BasePersistence implementation. > - Avoids requirements of open-ended multi-statement transactions. > > This approach doesn't necessitate model changes. My understanding is that > we've agreed on this implementation in its current state (as Dimitri > approved it), and the JDBCImpl will simply implement BasePersistence [2] > and utilize AtomicMetaStoreManager to eliminate the need for EclipseLink's > transactional operations. > > Regarding your comment: > > because IIRC the requirements that some people voiced is to keep the same > schema as the current Eclipselink one > > Yes, I'm aiming to maintain the same schema as EclipseLink in the new JDBC > implementation to facilitate easy migration. The entity table, grants > table, and all other schemas remain consistent. > > Regarding the JSONB suggestion, I wanted to gather the community's > thoughts. If we agree, we could incorporate it into both the JDBC and > EclipseLink implementations, or solely into the JDBC implementation. As > mentioned, JSONB would be beneficial for supporting new query patterns. > > Please share your thoughts on this. > > Best regards, > > Prashant Singh > > [1] > > https://github.com/apache/polaris/blob/main/polaris-core/src/main/java/org/apache/polaris/core/persistence/AtomicOperationMetaStoreManager.java > [2] > > https://github.com/apache/polaris/blob/main/polaris-core/src/main/java/org/apache/polaris/core/persistence/BasePersistence.java > > On Fri, Mar 21, 2025 at 5:23 AM Robert Stupp <sn...@snazy.de> wrote: > > > Hey Prashant, > > > > thanks for working on this! > > > > I'm a bit confused about "non transactional Postgres" - how's PG not > > transactional? If so (like one DML per tx), how is consistency > > guaranteed, because IIRC the requirements that some people voiced is to > > keep the same schema as the current Eclipselink one. I've been a bit > > confused whether people use that one or not, because different people > > tell different things. Can you shed some light on that? I mean, if EL > > isn't used and there are better ways, why should we keep EL at all? > > > > From my experience (via #1189), it's not possible to just issue one DML > > per PG tx and guarantee consistency with the data model defined by EL. > > So I suspect you're working on a different database schema. If so, how > > does it work? > > > > Can you share your design and work with everybody here? > > > > Robert > > > > [1] https://github.com/apache/polaris/pull/1189 > > > > On 20.03.25 19:21, Prashant Singh wrote: > > > Hey folks, > > > I am presently working on implementing non transactional postgres > > > implementation using jdbc. in the course of that i noticed in the > entity > > > table that the column type of the properties / internal properties > column > > > is TEXT and not JSONB, I dug a bit into the history of this, seems like > > we > > > did wanted to have JSONB but ended up implementing it as TEXT (may to > > > support more relational DB's which doesn't support JSONB) . > > > > > > JSONB has a couple of benefits, some of the noticeable ones are that it > > > lets you define indexes on them [1], which would help us in supporting > > some > > > different query patterns as we start supporting new entities. Until the > > > separate table per entity discussion is finalized, this could be > helpful > > as > > > with supporting the new query pattern. For ex policy need to presently > do > > > client side filtering as policyType which is a present inside > properties, > > > it needs to be parsed in client side and then do filtering at client > end. > > > > > > Please let me know your thoughts considering the above, happy to put > out > > a > > > PR for the same. > > > I am happy to handle this in my JDBC impl as this is anyway a user > would > > be > > > consciously choosing it over eclipse impl. I would love to know your > > > thoughts if we need to handle it in eclipse link impl as well. > > > > > > > > > Best, > > > Prashant > > > > > > [1] > > > > > > https://scalegrid.io/blog/using-jsonb-in-postgresql-how-to-effectively-store-index-json-data-in-postgresql/ > > > > > -- > > Robert Stupp > > @snazy > > > > >
