Hi folks, Thanks everyone who joined the Community Meeting about Persistence Improvements Proposal.
Here's the record: https://drive.google.com/file/d/1I26D1teNMdhhLe-o7ymG146AQ3GWU5zO/view?usp=sharing Sorry, I had to drop before the end due to another meeting. I would like to use this email for a sum-up and proposal to move forward. 1. Abstract The initial goal is to be able to support new storage backends (RDBMS and NoSQL), focusing on the persistence layer and without changing the Polaris entities/model. That's the general goal. 2. Approved changes/proposals To achieve the goal from 1, we identified three action items: a. Refactor the persistence "core" layer (PolarisMetaStoreManager, PolarisMetaStoreSession, ...) in order to be able to support SQL/NoSQL backends via interfaces. b. Provide JDBC (PostgreSQL) backend without EclipseLink using the interfaces from (a). c. Provide MongoDB backend using the interfaces from (a). The focus for Polaris 1.0.0 is a and b (c would be good to have but not a blocker). 3. Current status Dennis worked on (a) and it's now merged. We have basically two "paths" to implement a persistence backend. Prashanth is working on the JDBC backend based on (a). Robert is working on the MongoDB backend based on (a). However, several questions popped up during the meeting yesterday: - About (2.a), can we clarify how to create a backend ? Which path should be used ? What should the contributor do to implement a backend ? - About (2.c), the current proposal/PR is larger than just the MongoDB support: it introduces a NessieAdapter, able to support other backends (DynamoDB, etc). The question was: it's more than just MongoDB support, "pushing" for a new data model. 4. Next steps I propose to split the next steps in two parts: a. Short term (e.g. for 1.0.0 release), I propose to move forward according to the plan we discussed a while ago (1). It means, with 2.a now merged, 2.b and 2.c should *only* focus on JDBC and MongoDB (nothing else). b. Regarding the (c) and NessieAdapter, I think it needs more discussion with the community about that, especially a discussion about the Polaris model and also if NessieAdapterMetaStoreManager can replace the MetaStoreManager. I'm concerned that we have different paths/approaches here: how to test, maintain, verify ? What message do we send to persistence backend contributors ? We need a clear discussion and consensus there. Thoughts ? Regards JB
