Re: Catalog Federation API Spec updates - adding ConnectionConfigInfo

Thanks for pushing this forward, Dennis! For visibility: I approved the PR on GitHub. If anyone has any comments or concerns that we did not cover so far, please reply here or on the PR. As a follow-up, I guess we need a separate discussion about handling secrets in Polaris in general, which wil

Re: NoSQL database agnostic persistence

I'm starting to open some small, isolated PRs. The first ones are https://github.com/apache/polaris/pull/1229 and https://github.com/apache/polaris/pull/1230 - those don't change anything to the existing code base, but help reducing the size of the big https://github.com/apache/polaris/pull/11

Re: Clear separation of REST APIs

Thanks Robert for bringing this up! I see we mentioned two problems in this thread: 1) polaris-catalog-service.yaml contains reference to all catalog APIs including the ones for Iceberg and our Polaris specific APIs, which could be confusing to the users or new developers. 2) Polaris specific

Re: Polaris benchmarks proposal

Hi Ajantha, That's a good request. Imho, right now, before distributing any artifact (either on nightly build space https://nightlies.apache.org/), I prefer to have it "good enough" from a "legal" standpoint (e.g. LICENSE/NOTICE). I'm almost done about that for all artifacts (jar and distributio

Re: Catalog Federation API Spec updates - adding ConnectionConfigInfo

Thanks for the comments! Yes I agree we'll probably want to adjust the API, and we can definitely try to start with some reasonable placeholder around auth/secrets to unblock various parallel streams of work on federation while keeping an eye on how different secrets/auth models can be accommodated

Re: Clear separation of REST APIs

Thanks for the discussion so far! 'iceberg-rest-catalog-open-api.yaml' is declared to be a 1:1 copy of > Iceberg's v.1.7.1 'open-api/rest-catalog-open-api.yaml', but in fact it > has already diverged beyond just code formatting. The iceberg-rest-catalog-open-api.yaml file should indeed be a 1:1 c

Re: Polaris benchmarks proposal

> I cannot think of any issue with storing that code in the polaris-tools repository. While contributing the `catalog migrator tool` to `polaris-tools`, I encountered a challenge because this external repository needs to depend on Apache Polaris jars, which haven't been published yet by Apache Pol

Re: NoSQL database agnostic persistence

I think what you are referring to is the "number of tables per namespace" property. See the binary tree example in the docs where, after a binary tree of namespaces, 5 tables are created in each nam

Re: Polaris benchmarks proposal

On Wed, Mar 19, 2025 at 4:53 PM Jean-Baptiste Onofré wrote: > Hi Pierre > > Thanks ! > > I have a general comment: do we want the benchmark tool as part of > Polaris "core" repo or on polaris-tools ? > As we can consider this as a benchmark "tool", maybe it makes sense to > host it in https://git

Re: NoSQL database agnostic persistence

We could just keep the exact same total number of entities, so 65 catalogs, 10 namespaces each with 100 Tables and Views. Or you can scale it up if you want. The insights I'm particularly interested in are the dynamics of listing and accessing tables in a namespace with multiple entries. The areas

[Discuss] Change EntityTable properties and internal properties from TEXT to JSONB

Hey folks, I am presently working on implementing non transactional postgres implementation using jdbc. in the course of that i noticed in the entity table that the column type of the properties / internal properties column is TEXT and not JSONB, I dug a bit into the history of this, seems like we

Re: NoSQL database agnostic persistence

IMO we should test the s..t out of the implementation to ensure that it's stable and reliable and correct. Once that's done, we can certainly have some "realistic reference benchmark" with defined, different and concurrent user stories. That's definitely possible with Gatling. On 20.03.25 14