Re: [VOTE] Add 'inherited' and 'namespaces' Fields to GetApplicablePolicies API Response

[Honah J.]

Thank you for your comment. Based on our discussion, I renamed the new
"namespaces" field to "namespace-path". This change better reflects that
the field identifies a single namespace and differentiates it from the
namespace parameter, which is a single string.

Please continue to vote.

Thanks,
Honah(Jonas)

On Fri, Apr 4, 2025 at 4:42 PM Dmitri Bourlatchkov  wrote:

> +0 the PR looks ok to me, but I made a non-critical comment in GH.
>
> Thanks,
> Dmitri.
>
> On Fri, Apr 4, 2025 at 5:23 PM Honah J.  wrote:
>
> > Hi folks,
> >
> > I'd like to raise a vote for adding `inhertied` and `namespaces` fields
> to
> > GetApplicablePolicies' response. These fields give users more information
> > about whether the effective policy is inherited from its parent and the
> > namespaces where the effective policy is located. You can find more
> details
> > in the discussion thread:
> >
> > https://lists.apache.org/thread/z2lh1cqjcznc1qvtwy46bq18p6psz5bn
> >
> > This vote will be open for at least 72 hours.
> >
> > [ ] +1 Merge https://github.com/apache/polaris/pull/1277
> > [ ] +0
> > [ ] -1 Do not merge because...
> >
>

Re: [DISCUSS] Release cycle for Spark Client

[Dmitri Bourlatchkov]

Hi Yun,

Your proposal LGTM.

However, regarding compatibility, I think this information has to be
tracked regardless of the release cycle, because users can mix different
client / server versions in their environments.

Cheers,
Dmitri.

On Tue, Mar 25, 2025 at 5:01 PM yun zou  wrote:

> Hi Team,
>
> Given that we are now introducing Spark Client, one thing we need to decide
> is the release cycle for the Spark Plugin.
>
> I propose to bundle the client release with Polaris main release like
> Iceberg. In that way, users will be able to get the client support for new
> APIs in the same release, and version compatibility is also implicitly
> implied in the release.
>
> An alternative is to release the client independently like
> spark-cassandra-connector. In that case, the client will not have to do
> extra release if there is no server API change. The drawback is that the
> client release will need to start after the server release is finished in
> case there are new changes, and extra Compatible Version information needs
> to be published to help users understand the compatibility.
>
> Please let me know about your thoughts  on the client release cycles.
>
> Best Regards,
> Yun
>

Re: [VOTE] Add 'inherited' and 'namespaces' Fields to GetApplicablePolicies API Response

[Dmitri Bourlatchkov]

Thanks for the update Honah!

Upgrading my vote to +1

Cheers,
Dmitri.

On Fri, Apr 4, 2025 at 7:54 PM Honah J.  wrote:

> Thank you for your comment. Based on our discussion, I renamed the new
> "namespaces" field to "namespace-path". This change better reflects that
> the field identifies a single namespace and differentiates it from the
> namespace parameter, which is a single string.
>
> Please continue to vote.
>
> Thanks,
> Honah(Jonas)
>
> On Fri, Apr 4, 2025 at 4:42 PM Dmitri Bourlatchkov 
> wrote:
>
> > +0 the PR looks ok to me, but I made a non-critical comment in GH.
> >
> > Thanks,
> > Dmitri.
> >
> > On Fri, Apr 4, 2025 at 5:23 PM Honah J.  wrote:
> >
> > > Hi folks,
> > >
> > > I'd like to raise a vote for adding `inhertied` and `namespaces` fields
> > to
> > > GetApplicablePolicies' response. These fields give users more
> information
> > > about whether the effective policy is inherited from its parent and the
> > > namespaces where the effective policy is located. You can find more
> > details
> > > in the discussion thread:
> > >
> > > https://lists.apache.org/thread/z2lh1cqjcznc1qvtwy46bq18p6psz5bn
> > >
> > > This vote will be open for at least 72 hours.
> > >
> > > [ ] +1 Merge https://github.com/apache/polaris/pull/1277
> > > [ ] +0
> > > [ ] -1 Do not merge because...
> > >
> >
>

[DISCUSS] Secrets Management for Catalog Federation

[Dennis Huo]

Hello!

I wanted to continue discussion that had started during review of the
initial Catalog Federation API spec (
https://github.com/apache/polaris/pull/1026) regarding how we want to
handle secrets.

I drafted this document describing how secrets management fits in to
Catalog Federation, while extrapolating some interfaces that can define
Polaris management of "user secrets" in a general way beyond federation as
well:

https://docs.google.com/document/d/1JPNx5vL4vM8DqwRwnBIPiQxwN4MXOdGx4Ki0j7vgwSM/edit?tab=t.0

One aspect incorporated in the design is the idea of independent trust
boundaries between each of:

- Secrets Manager (e.g. Vault, KMS, etc)
- Persistence Database (e.g. Postgres, MongoDB, etc)
- Service runtime (e.g. JVM running in k8s in a VM in cloud)

The gist of it is that regardless of how users specify secrets attached to
Polaris objects (such as an ExternalCatalog that is a federation facade),
we canonicalize an internal "UserSecretReference" to be used in internal
persistence objects (and internal persistence objects never contain a
member holding a plaintext secret).

The extensible UserSecretReference allows things like layered security and
encryption such that neither the external secrets manager nor the
persistence database alone store plaintext secrets in case either one is
compromised independently.

This PR is an end-to-end working MVP of Catalog Federation that includes an
in-memory implementation of the UserSecretsManager exercising the
layered-security flow; it may be helpful to see the secrets management in
the full context of the working feature to see how its pieces fit together:
https://github.com/apache/polaris/pull/1305

Would love to hear everyone's thoughts! Feel free to also comment in the
document directly.

Cheers,
Dennis

Re: [DISCUSS] Preparing 0.10.0 release including binary distributions

[Yufei Gu]

I'm OK with 0.10.0-preview or Robert's idea of with a "-beta" or
"-experimental" or 1.0.0-preview. We just need to give users a clear
message that it's not a stable release you can rely on.

Yufei


On Wed, Mar 19, 2025 at 7:25 AM Jean-Baptiste Onofré 
wrote:

> Thanks guys for the feedback.
>
> @Yufei, what do you think about 0.10.0 release then ?
>
> If we don't find a consensus on versioning, then, let's focus on the
> 1.0.0 release directly.
>
> I would like to remind the purpose of the 0.10.0/1.0.0-preview release
> purpose:
> - include binary distributions
> - include legal aspect for all artifacts
> - submit the release to the mentors and, if passed, to the IPMC
> The idea is to validate a release with binary distributions/artifacts
> before the 1.0.0 release.
>
> Regards
> JB
>
> On Wed, Mar 19, 2025 at 2:54 PM Dmitri Bourlatchkov 
> wrote:
> >
> > I'd be fine with 1.0.0-preview1 if we had a solid Persistence codebase.
> >
> > I do not think that 1.0.0 necessarily conveys the idea of stability in
> > runtime, but I do believe that it has strong connotations for semver.
> Given
> > that NoSQL persistence is not settled yet, I'd like to avoid making a 1.0
> > release because then we'd have to introduce major persistence changes in
> a
> > minor release, which IMHO does not align with semver concepts very well.
> >
> > ... and I guess we're not ready for 2.0 yet :)
> >
> > Cheers,
> > Dmitri.
> >
> > On Wed, Mar 19, 2025 at 8:38 AM Jean-Baptiste Onofré 
> > wrote:
> >
> > > Hi everyone,
> > >
> > > @Dmitri and @Robert, are you fine with 1.0.0-preview1 ?
> > >
> > > I would like to move forward on this front ;)
> > >
> > > Thanks,
> > > Regards
> > > JB
> > >
> > > On Tue, Mar 18, 2025 at 12:08 AM Yufei Gu 
> wrote:
> > > >
> > > > Agreed with JB that 1.0-pre makes sense. My concern with 0.10.0 is
> that
> > > it
> > > > could mislead users into thinking an arbitrary cut from main
> qualifies
> > > as a
> > > > stable release.
> > > >
> > > >
> > > > Yufei
> > > >
> > > >
> > > > On Mon, Mar 17, 2025 at 10:30 AM Jean-Baptiste Onofré <
> j...@nanthrax.net>
> > > > wrote:
> > > >
> > > > > It depends. For instance, Spark 4.0-preview started more than a
> year
> > > > > ago, and the scope changed.
> > > > > If we communicate clearly it's just a previous and the scope can
> still
> > > > > change, it's acceptable.
> > > > >
> > > > > I did the same on multiple projects: Camel 4.0.0.M1, M2, RC1, RC2,
> > > > > were different in content.
> > > > >
> > > > > I would separate the discussions in two parts:
> > > > > 1. Should we have a discussion about what will be included in 1.0 ?
> > > > > Maybe yes, based on what we have in the GitHub Milestone. I would
> > > > > propose to start a separate thread about that.
> > > > > 2. For preview release, as the idea is:
> > > > > 2.1. Do a preparation release from main, different from 0.9.0,
> > > > > including binary artifacts.
> > > > > 2.2. Verify all legal aspects and the release is OK for the IPMC
> > > > > So, 0.10.0 or 1.0.0-preview work. Personally, I consider
> 1.0.0-preview
> > > > > more meaningful because, without considering the scope/content,
> it's
> > > > > really what it is: a preview release to test our process and legal
> > > > > aspects.
> > > > >
> > > > > Regards
> > > > > JB
> > > > >
> > > > > On Mon, Mar 17, 2025 at 4:39 PM Dmitri Bourlatchkov <
> di...@apache.org>
> > > > > wrote:
> > > > > >
> > > > > > Using 1.0.0-preview1 implies the scope of 1.0 is well-defined...
> but
> > > my
> > > > > > impression is that it is not so.
> > > > > >
> > > > > > I think the 0.10.0 version is clear enough that it comes before
> 1.0
> > > and
> > > > > > does not have any implied scope.
> > > > > >
> > > > > > While 0.10.0 is in progress, I believe we need to review the
> scope
> > > of 1.0
> > > > > > as a community on the dev list. I might have missed previous
> > > discussions,
> > > > > > but I do not recall a consensus on what goes into 1.0 :)
> > > > > >
> > > > > > WDYT?
> > > > > >
> > > > > > Thanks,
> > > > > > Dmitri.
> > > > > >
> > > > > > On Mon, Mar 17, 2025 at 9:49 AM Jean-Baptiste Onofré <
> > > j...@nanthrax.net>
> > > > > > wrote:
> > > > > >
> > > > > > > Usually, at Apache, we have two kind of versioning for
> > > "pre-release":
> > > > > > > - 1.0.0.M1 and 1.0.0.RC1 (Apache Superset, Apache Camel, Apache
> > > Karaf,
> > > > > > > Apache Cassandra, ... used this versioning)
> > > > > > > - 1.0.0-preview1 (Apache Spark, Apache Flink, ... used this
> > > versioning)
> > > > > > >
> > > > > > > For "clarity" for our community and users, I propose to use
> Apache
> > > > > > > Polaris (incubating) 1.0.0-preview1.
> > > > > > >
> > > > > > > Any objections?
> > > > > > >
> > > > > > > Regards
> > > > > > > JB
> > > > > > >
> > > > > > > On Mon, Mar 17, 2025 at 2:37 PM Kamesh Sampath
> > > > > > >  wrote:
> > > > > > > >
> > > > > > > > Shall we name it like 1.0-pre? That aligns with common
> pattern
> > > across
> > > > > > > many opensourc

[Apache Polaris Meeting] Persistence Improvements Discussion

[Jean-Baptiste Onofré]

Hi folks,

As already proposed on another thread, and after our Community Meeting
yesterday, I would like to propose a Community Meeting to have an
update about the recent improvements we did (and still doing :)) on
Polaris Persistence.

I sent an invite for Thursday, March 27th, 9am PST.

Here's the public link for the Google Meet:
https://calendar.app.google/sbCnCTXYE3BbWkrS7

As usual, this meeting will be recorded and shared on the dev mailing list.

Thanks !
Regards
JB

Re: Rename table maintenance policy "snapshot-retention" to "snapshot-expiry"

[Dmitri Bourlatchkov]

For visibility: I posed my comment in GH (minor):
https://github.com/apache/polaris/pull/1284#discussion_r2022896891

Cheers,
Dmitri.

On Mon, Mar 31, 2025 at 2:55 PM Yufei Gu  wrote:

> Hi folks,
>
> I'm proposing to rename the table maintenance policy "snapshot-retention"
> to "snapshot-expiry". While "snapshot-retention" is a reasonable name, it's
> a bit too general. I believe "snapshot-expiry" more precisely captures the
> intent of the policy, which is specifically about removing expired
> snapshots.
>
> I think this is the right time to make the change before we include the
> policy schemas in an upcoming release.
>
> Here’s the PR for reference: https://github.com/apache/polaris/pull/1284
>
> If you have any concerns or suggestions for a better name, feel free to
> share your thoughts.
>
> Yufei
>

Re: Polaris benchmarks proposal

[Robert Stupp]

Having benchmark results against individual commits is a great thing to 
have.


The small GH hosted runners however are not suitable for 
deterministic/comparable results.


It would be possible though, if the hardware (or bare-metal compute 
instances in the cloud) is available to the project. I suspect there's 
nothing that would block the project from getting this sponsored, am I 
right JB?



On 19.03.25 17:09, Prashant Singh wrote:

Thank you so much for the benchmarks !
+1, having benchmark results committed, it will help catch any degradation
/ correctness issue that can creep in !
equivalent to golden files of tpc-ds / tpc-h in spark repo.

Best,
Prashant Sungh

On Wed, Mar 19, 2025 at 8:53 AM Russell Spitzer 
wrote:


I think having a tool like this is a great idea. Would we be able to host
the results over time as well? Like an official build run that triggers on
a daily basis?

On Wed, Mar 19, 2025 at 10:07 AM Pierre Laporte 
wrote:


Hi

I have been working on a set of benchmarks for Polaris [1] and would like
to contribute them to the project.  I have opened a PR with the code, in
case anybody is interested.

The benchmarks are written using Gatling.  The core design decision
consists in building a procedural dataset, loading it to Polaris, and

then

reusing it for all subsequent benchmarks.  The procedural aspect makes it
possible to deterministically regenerate the same dataset at runtime over
and over, without having to store the actual data.

With this, it is trivial to generate large number of Polaris entities.
Typically, I used this to benchmark the NoSQL persistence implementation
with 65k namespaces, 65k tables and 65k views.  Increasing that to

millions

would only require a one parameter change.  Additionally, the dataset
currently includes property updates for namespaces, tables and views,

which

can quickly create hundreds of manifests.  This can be useful for table
maintenance testing.

Three benchmarks have been created so far:

- A benchmark that populates an empty Polaris server with a dataset

that

have predefined attributes
- A benchmark that issues only read queries over that dataset
- A benchmark that issues read and write queries (entity updates) over
that dataset, with a configurable read/write ratio

The benchmarks/README.md contains instructions to build and run the
benchmarks, as well as to describe the kind of dataset that should be
generated.

As with every Gatling benchmark, an HTML page is generated with

interactive

charts showing query performance over time, response time percentiles,
etc...

I would love to head your feedback on it.

Pierre

[1] https://github.com/apache/polaris/pull/1208
--

Pierre Laporte
@pingtimeout 
pie...@pingtimeout.fr
http://www.pingtimeout.fr/


--
Robert Stupp
@snazy

Re: [DISCUSS] Preparing 0.10.0 release including binary distributions

[Jean-Baptiste Onofré]

Hi Yufei,

Thanks for your reply!

I propose to use 0.10.0-beta (as consensus).

I'm moving forward on the PR about LICENSE/NOTICE and checking the
artifacts. I will create this milestone on GitHub.

Thanks,
Regards
JB

On Thu, Mar 20, 2025 at 8:04 PM Yufei Gu  wrote:
>
> I'm OK with 0.10.0-preview or Robert's idea of with a "-beta" or
> "-experimental" or 1.0.0-preview. We just need to give users a clear
> message that it's not a stable release you can rely on.
>
> Yufei
>
>
> On Wed, Mar 19, 2025 at 7:25 AM Jean-Baptiste Onofré 
> wrote:
>
> > Thanks guys for the feedback.
> >
> > @Yufei, what do you think about 0.10.0 release then ?
> >
> > If we don't find a consensus on versioning, then, let's focus on the
> > 1.0.0 release directly.
> >
> > I would like to remind the purpose of the 0.10.0/1.0.0-preview release
> > purpose:
> > - include binary distributions
> > - include legal aspect for all artifacts
> > - submit the release to the mentors and, if passed, to the IPMC
> > The idea is to validate a release with binary distributions/artifacts
> > before the 1.0.0 release.
> >
> > Regards
> > JB
> >
> > On Wed, Mar 19, 2025 at 2:54 PM Dmitri Bourlatchkov 
> > wrote:
> > >
> > > I'd be fine with 1.0.0-preview1 if we had a solid Persistence codebase.
> > >
> > > I do not think that 1.0.0 necessarily conveys the idea of stability in
> > > runtime, but I do believe that it has strong connotations for semver.
> > Given
> > > that NoSQL persistence is not settled yet, I'd like to avoid making a 1.0
> > > release because then we'd have to introduce major persistence changes in
> > a
> > > minor release, which IMHO does not align with semver concepts very well.
> > >
> > > ... and I guess we're not ready for 2.0 yet :)
> > >
> > > Cheers,
> > > Dmitri.
> > >
> > > On Wed, Mar 19, 2025 at 8:38 AM Jean-Baptiste Onofré 
> > > wrote:
> > >
> > > > Hi everyone,
> > > >
> > > > @Dmitri and @Robert, are you fine with 1.0.0-preview1 ?
> > > >
> > > > I would like to move forward on this front ;)
> > > >
> > > > Thanks,
> > > > Regards
> > > > JB
> > > >
> > > > On Tue, Mar 18, 2025 at 12:08 AM Yufei Gu 
> > wrote:
> > > > >
> > > > > Agreed with JB that 1.0-pre makes sense. My concern with 0.10.0 is
> > that
> > > > it
> > > > > could mislead users into thinking an arbitrary cut from main
> > qualifies
> > > > as a
> > > > > stable release.
> > > > >
> > > > >
> > > > > Yufei
> > > > >
> > > > >
> > > > > On Mon, Mar 17, 2025 at 10:30 AM Jean-Baptiste Onofré <
> > j...@nanthrax.net>
> > > > > wrote:
> > > > >
> > > > > > It depends. For instance, Spark 4.0-preview started more than a
> > year
> > > > > > ago, and the scope changed.
> > > > > > If we communicate clearly it's just a previous and the scope can
> > still
> > > > > > change, it's acceptable.
> > > > > >
> > > > > > I did the same on multiple projects: Camel 4.0.0.M1, M2, RC1, RC2,
> > > > > > were different in content.
> > > > > >
> > > > > > I would separate the discussions in two parts:
> > > > > > 1. Should we have a discussion about what will be included in 1.0 ?
> > > > > > Maybe yes, based on what we have in the GitHub Milestone. I would
> > > > > > propose to start a separate thread about that.
> > > > > > 2. For preview release, as the idea is:
> > > > > > 2.1. Do a preparation release from main, different from 0.9.0,
> > > > > > including binary artifacts.
> > > > > > 2.2. Verify all legal aspects and the release is OK for the IPMC
> > > > > > So, 0.10.0 or 1.0.0-preview work. Personally, I consider
> > 1.0.0-preview
> > > > > > more meaningful because, without considering the scope/content,
> > it's
> > > > > > really what it is: a preview release to test our process and legal
> > > > > > aspects.
> > > > > >
> > > > > > Regards
> > > > > > JB
> > > > > >
> > > > > > On Mon, Mar 17, 2025 at 4:39 PM Dmitri Bourlatchkov <
> > di...@apache.org>
> > > > > > wrote:
> > > > > > >
> > > > > > > Using 1.0.0-preview1 implies the scope of 1.0 is well-defined...
> > but
> > > > my
> > > > > > > impression is that it is not so.
> > > > > > >
> > > > > > > I think the 0.10.0 version is clear enough that it comes before
> > 1.0
> > > > and
> > > > > > > does not have any implied scope.
> > > > > > >
> > > > > > > While 0.10.0 is in progress, I believe we need to review the
> > scope
> > > > of 1.0
> > > > > > > as a community on the dev list. I might have missed previous
> > > > discussions,
> > > > > > > but I do not recall a consensus on what goes into 1.0 :)
> > > > > > >
> > > > > > > WDYT?
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Dmitri.
> > > > > > >
> > > > > > > On Mon, Mar 17, 2025 at 9:49 AM Jean-Baptiste Onofré <
> > > > j...@nanthrax.net>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Usually, at Apache, we have two kind of versioning for
> > > > "pre-release":
> > > > > > > > - 1.0.0.M1 and 1.0.0.RC1 (Apache Superset, Apache Camel, Apache
> > > > Karaf,
> > > > > > > > Apache Cassandra, ... used this versioning)
> > > > > > > > - 1.0.0

[VOTE] Add 'inherited' and 'namespaces' Fields to GetApplicablePolicies API Response

[Honah J.]

Hi folks,

I'd like to raise a vote for adding `inhertied` and `namespaces` fields to
GetApplicablePolicies' response. These fields give users more information
about whether the effective policy is inherited from its parent and the
namespaces where the effective policy is located. You can find more details
in the discussion thread:

https://lists.apache.org/thread/z2lh1cqjcznc1qvtwy46bq18p6psz5bn

This vote will be open for at least 72 hours.

[ ] +1 Merge https://github.com/apache/polaris/pull/1277
[ ] +0
[ ] -1 Do not merge because...

Re: [Discuss] Add 'inherited' and 'namespaces' Fields to GetApplicablePolicies API Response

[Yufei Gu]

Thanks for bringing this up, Jonas!

The changes make sense to me!

Yufei


On Sat, Mar 29, 2025 at 5:44 PM Honah J.  wrote:

> Hi folks,
>
> I'd like to propose a change to our existing GetApplicablePolicies API
> endpoint (GET /v1/{prefix}/applicable-policies) and gather your feedback.
>
> Currently, this endpoint retrieves policies applicable to a given target
> (e.g., table or view), including those inherited from parent entities. For
> example, the UI may utilize this endpoint to identify and display policies
> like data compaction or metadata retention for a specific table.
>
> The existing API response includes the following fields for each policy:
>
>- name
>- type
>- inheritable
>- description
>- content
>- version
>
> However, these fields are insufficient for scenarios where users need to
> override a policy. Consider the following scenario:
>
> Suppose a data-compaction policy (P1) is assigned to namespace NS1, causing
> all tables within NS1 to inherit this policy. If a user wants to override
> the data-compaction setting specifically for a table (TB1) within NS1, they
> currently can only rely on the response from calling getApplicablePolicies
> on TB1. However, this response does not indicate:
>
>- The namespace where the applicable policy resides.
>- Whether the policy is directly attached to TB1 or inherited from a
>parent namespace.
>
> As a result, users cannot determine if modifying the current policy
> directly would inadvertently impact other tables. They also lack sufficient
> information to update the policy.
>
> To resolve this, I propose adding two additional fields to each applicable
> policy in the API response:
>
>-
>
>inherited: A boolean indicating if the policy is inherited from a parent
>entity.
>-
>
>namespaces: A hierarchical list showing the policy’s namespaces, ordered
>from the highest-level namespace down to the immediate parent.
>
> These fields enable users to decide whether to attach a new policy directly
> to TB1 or to update an existing policy without affecting settings on other
> tables
>
> Here is the draft PR for the proposed spec change:
> https://github.com/apache/polaris/pull/1277
>
> Looking forward to your feedback and suggestions!
>
> Best regards,
>
> Honah (Jonas)
>

Re: [VOTE] Add 'inherited' and 'namespaces' Fields to GetApplicablePolicies API Response

[Dmitri Bourlatchkov]

+0 the PR looks ok to me, but I made a non-critical comment in GH.

Thanks,
Dmitri.

On Fri, Apr 4, 2025 at 5:23 PM Honah J.  wrote:

> Hi folks,
>
> I'd like to raise a vote for adding `inhertied` and `namespaces` fields to
> GetApplicablePolicies' response. These fields give users more information
> about whether the effective policy is inherited from its parent and the
> namespaces where the effective policy is located. You can find more details
> in the discussion thread:
>
> https://lists.apache.org/thread/z2lh1cqjcznc1qvtwy46bq18p6psz5bn
>
> This vote will be open for at least 72 hours.
>
> [ ] +1 Merge https://github.com/apache/polaris/pull/1277
> [ ] +0
> [ ] -1 Do not merge because...
>