Re: [PROPOSAL] Add Data Lake operational metrics to Polaris

suited for large tables. 5. This proposal does not cover Iceberg commit and scan reports -- Pierre On Mon, Sep 15, 2025 at 8:58 PM Pierre Laporte wrote: > > > On Mon, Sep 15, 2025 at 8:17 PM Yufei Gu wrote: > >> > From my perspective, Polaris should compute the aggregati

Re: [PROPOSAL] Add Data Lake operational metrics to Polaris

On Mon, Sep 15, 2025 at 8:17 PM Yufei Gu wrote: > > From my perspective, Polaris should compute the aggregations it needs > (like file size distributions) > > That’d be a pretty big perf hit on Polaris itself, unless you mean Polaris > in the broader sense, including peripheral services like TMS.

Re: [PROPOSAL] Add Data Lake operational metrics to Polaris

ndpoint, and > then figure out a persistence strategy as we go, for example, leveraging > time-series databases to better serve historical data. > > > Yufei > > > On Fri, Sep 5, 2025 at 1:16 PM Pierre Laporte > wrote: > > > I was definitely not aware of that

Re: [PROPOSAL] Add Data Lake operational metrics to Polaris

Looking forward to discussing this more ! > > Best, > Prashant Singh > > > On Fri, Sep 5, 2025 at 5:03 AM Pierre Laporte > wrote: > > > Thanks for the feedback, Prashant > > > > As far as I can tell, we could use the Iceberg Metrics Reporting for >

Re: [PROPOSAL] Add Data Lake operational metrics to Polaris

/main/java/org/apache/iceberg/metrics/CommitMetrics.java > > > are already available but at this point we don't persist them and hence > they are lost, there has been a request for this in Polaris slack too ! > My recommendations would start from here ! > > Best, > Prashant Si

[PROPOSAL] Add Data Lake operational metrics to Polaris

Hi folks, I would like to propose the addition of a component to Polaris that would build and maintain operational metrics for the Data Lake tables and views. The main idea is that, if those metrics can be shared across multiple Table Management Services and/or other external services, then it wou

Re: [VOTE] Release Apache Polaris 1.1.0-incubating (rc0)

Just a heads up, I was able to reproduce the issue when I switched to Docker Desktop. But when I tested the build with OrbStack and Podman Desktop, it went well. So it seems that the issue happens only on MacOs+Docker Desktop. Adding that in #2501. I also don't think we should consider this as

Re: [VOTE] Release Apache Polaris 1.1.0-incubating (rc0)

> > > > ``` > > > > > Task :polaris-runtime-service:intTest > > > > > > > > RestCatalogKeycloakFileIT > initializationError FAILED > > > > org.junit.jupiter.api.extension.ParameterResolutionException at > > > > ArrayLis

Re: [VOTE] Release Apache Polaris 1.1.0-incubating (rc0)

sion.ParameterResolutionException at > > > ArrayList.java:1596 > > > Caused by: java.lang.NullPointerException at Objects.java:259 > > > ``` > > > > > > > > > On Tue, Sep 2, 2025 at 4:46 AM Pierre Lap

Re: [VOTE] Release Apache Polaris 1.1.0-incubating (rc0)

+1 (non-binding) Verified: * Signatures and checksums * Polaris starts both in-memory and with JDBC persistence * Sanity-check read and write benchmarks complete successfully -- Pierre On Mon, Sep 1, 2025 at 4:28 PM Alexandre Dutra wrote: > +1 (nb) > > Verified: > > - Signatures & checksums

Re: 1.0.1 and versioned docs

+1 for that logic with one caveat: if we do follow a release train model, in the best case, we would cut a new `X.Y.0` release every month. And that might make the docs disappear quickly. Example: assume we release 1.1.0 in August and do not have to do any patch release. It follows that we relea

Re: [DISCUSS] Require changelog updates within applicable PR

Thanks for your feedback, Yufei. Let me clarify a bit the proposal On Fri, Aug 22, 2025 at 2:47 AM Yufei Gu wrote: > I'm not a big fan of it, as I'm not sure how we execute it effectively. > 1. How do we define "applicable" PR? > If we want to be exhaustive, indeed, we can come up with a list

Re: [DISCUSS] Labels for GitHub issues + PRs

+1 for removing unnecessary labels. As for whether we keep the blocker label, I am +0 on keeping it. I am ok if it stays for the time being, but I don't think we need it anymore. Taking the 1.1.0 as an example, we are not using the blocker label. But still, we are discussing the possibility to s

Re: [DISCUSSION] Integrations points in Polaris

It would probably be good to separate the API modules, which define what other systems can consume, from SPI modules, which define what other systems can _extend_. Those are two different concerns and could require two distinct sets of dependencies. However I do not think we should consider the c

[DISCUSS] Require changelog updates within applicable PR

Hi folks Creating this thread to avoid hijacking the 1.1.0 release thread. The topic of keeping a change log for breaking changes was already discussed on the dev mailing list [1]. But one point related to ownership of CHANGELOG.md consistency was not addressed [2]. And I would like to revive t

Re: [RELEASE] 1.1.0-incubating release preparation

I just opened https://github.com/apache/polaris/pull/2406 with the list of new features that were added in main since the 1.0.x release branch was cut. Would it be possible to get a review to ensure I did not miss any? Note that this only contains features, and not fixes. -- Pierre On Wed, Au

Re: [DISCUSS] Apache Polaris (incubating) 1.1.0 release mid August ?

Hi folks The more I work on release automation, the more I realize certain parts are likely not described yet. It would be good if I could either perform the release (or shadow the release manager). So if nobody has volunteered yet, I would like to volunteer for the 1.1.0 release manager role.

Re: [VOTE] Release Apache Polaris 1.0.1-incubating (RC0)

Hi folks Checks passed: * Verified links for binaries and helm chart * Verified signatures and checksums * Verified that Polaris starts using binaries and Docker image * Ran simple test with catalog, namespace, table and view creation There is one point I am not sure about. The Helm chart curren

Re: [PROPOSAL] Release Apache Polaris 1.0.1-incubating

JB would it be possible to join you for the release as an observer? I got a few code review comments that suggest the release guide is not 100% up to date and I would like to compare it with what is done for 1.0.1 so that I can update it. -- Pierre On Thu, Jul 24, 2025 at 5:39 AM Jean-Baptiste

Release automation scripts

Hello folks I just published a new PR (#2156) so that Polaris has semi-automated releases [1]. This work is a 1:1 mapping of the release guide into bash scripts, with a couple of additional checks. This is intended to facilitate reviews and so that i

Re: [ANNOUNCE] Apache Polaris 1.0.0-incubating

Congratulations everyone, this is a huge milestone ! -- Pierre On Thu, Jul 10, 2025 at 5:59 PM Dmitri Bourlatchkov wrote: > Great news! Thanks for driving the release, Yufei! > > Cheers, > Dmitri. > > On Wed, Jul 9, 2025 at 8:42 PM Yufei Gu wrote: > > > The Apache Polaris team is pleased to

Re: [DISCUSS] Gradle and Maven runner plugins

On Tue, Jun 17, 2025 at 1:42 AM Yufei Gu wrote: > Thanks for the summary. > > * Given that a solution based on Gradle ExecFork or Gretty would only serve > > Polaris devs, and would not be usable by users, it does not seems to be a > > good fit for the overarching goal. > > Could you clarify the

Re: [DISCUSS] Gradle and Maven runner plugins

/pull/18 > [2]. https://github.com/psxpaul/gradle-execfork-plugin > [3]. https://github.com/gretty-gradle-plugin/gretty > > > Yufei > > > On Mon, Jun 2, 2025 at 12:44 AM Pierre Laporte > wrote: > > > On Thu, May 29, 2025 at 7:40 PM Yufei Gu wrote: > >

Re: [DISCUSS] PR #1724 – Directory & package refactor for persistence modules

Could you add a README file at the root of the `persistence/` directory so that it is clear that eclipselink is an exception to the new structure? Otherwise, there will be two persistence implementations in Polaris with different package naming schemes, which can add confusion from an external pers

Re: [DISCUSS] Gradle and Maven runner plugins

On Thu, May 29, 2025 at 7:40 PM Yufei Gu wrote: > I'm supportive if it adds value to both Polaris devs and users. Ok thanks, I am glad we have agreement on the initiative, even if we still have to discuss the implementation details. But, I don't think the Polaris project is in a position to pr

Re: [DISCUSS] Gradle and Maven runner plugins

pper, or even a simple > Gradle exec hook would cover the use cases? > > Yufei > > > On Tue, May 27, 2025 at 11:17 AM Pierre Laporte > wrote: > > > Hi all > > > > The PR polaris-tools#18 ( > https://github.com/apache/polaris-tools/pull/18/) > > aims

[DISCUSS] Gradle and Maven runner plugins

Hi all The PR polaris-tools#18 (https://github.com/apache/polaris-tools/pull/18/) aims at adding the ability to run temporary Polaris servers using Maven and Gradle. It is a successor to https://github.com/apache/polaris/pull/785 that was initially proposed for the apache/polaris repository. The

Re: [VOTE] Release Apache Polaris 0.10.0-beta-incubating (rc3)

I believe the _helpers.tpl file is mentioned in LICENSE ( https://github.com/apache/polaris/blob/apache-polaris-0.10.0-beta-incubating-rc3/LICENSE#L317). Am I missing something? -- Pierre On Wed, May 21, 2025 at 6:14 AM Jean-Baptiste Onofré wrote: > Thanks Ryan > > If the helm chart tarball n

Re: [DISCUSS] Closing or rescoping reported EntityCache issue currently marked as 1.0 blocker https://github.com/apache/polaris/issues/761

On Fri, May 2, 2025 at 7:01 AM Dennis Huo wrote: > Hello all, > > Following up from today's community sync where we were discussing 1.0 > blockers, I just wanted to finalize discussion on > https://github.com/apache/polaris/issues/761 to check if everyone's > aligned > on the current state of thi

Re: [VOTE] Release Apache Polaris 0.10.0-beta-incubating

Also -1 (non-binding) The postgresql driver is correctly bundled in the server distribution. But it is missing from the the admin tool distribution. See below ``` $ unzip -l ~/Downloads/polaris-quarkus-admin-0.10.0-beta-incubating.zip | grep postgres $ $ unzip -l ~/Downloads/polaris-quarkus-ser

Re: Next steps for Polaris benchmarks

On Mon, Apr 21, 2025 at 6:34 PM Yufei Gu wrote: > Thanks Pierre for driving this. The plan sounds good to me! A side > question, are we planning to make a benchmark pipeline against the main > branch? I think it would be good to setup a pipeline against `main`, yes. Although I am not sure whic

Next steps for Polaris benchmarks

Hello folks During the community sync, there as an item for benchmarks next step but we could not get to it. I don't think we need to wait for the next community sync to start the discussion, so here we go. I have a couple of tasks on my todo list for Polaris benchmarks. And I would like to sha

Re: Pull requests against polaris-tools

not, but wanted to double check. > > Regards > JB > > On Wed, Apr 16, 2025 at 6:31 PM Pierre Laporte > wrote: > > > > Hi folks > > > > I am going to open a few PR against the polaris-tools repository to > improve > > benchmarks, especially rega

Pull requests against polaris-tools

Hi folks I am going to open a few PR against the polaris-tools repository to improve benchmarks, especially regarding update-related workloads. I opened a new PR but noticed that, unlike with the PR against `apache/polaris`, nobody is automatically added as reviewer when a pull request is created

Re: Discussion: Re-evaluating Realm Modeling in Polaris

Hi Prashant I guess the answer will depend on how easy it should be for Polaris to support multi-tenancy. A separate database per realm would allow administrators to limit the amount of resources that a realm can consume (e.g. the maximum number of database connections). Indeed, it would be one

Re: Polaris benchmarks proposal

f a -.1. The > reason I > > > am > > > > > still negative is that inclusion of the benchmarks into the project > > > isn't > > > > > just about utility to the project, but about whether the community > > > should > > > &

EntityCache race conditions and JCStress in Polaris

Hello all In https://github.com/apache/polaris/issues/761, there is a discussion on whether the EntityCache is thread-safe, prone to race conditions, inconsistencies, etc... To determine whether these issues are real, I worked on JCStress [1] tests. JCStress is a testing framework developed by O

Re: Polaris benchmarks proposal

gt; > > > > I cannot think of any issue with storing that code in the > > polaris-tools > > > > repository. > > > > > > > > While contributing the `catalog migrator tool` to `polaris-tools`, I > > > > encountered a challenge

Re: Polaris benchmarks proposal

On Wed, Mar 19, 2025 at 4:53 PM Jean-Baptiste Onofré wrote: > Hi Pierre > > Thanks ! > > I have a general comment: do we want the benchmark tool as part of > Polaris "core" repo or on polaris-tools ? > As we can consider this as a benchmark "tool", maybe it makes sense to > host it in https://git

Re: NoSQL database agnostic persistence

? Could we run some of those as well? It would be great to have > something with closer to a 1 Namspace to 100 tables sort of layout. > > On Wed, Mar 19, 2025 at 3:06 PM Pierre Laporte > wrote: > > > Just a heads up, I updated the report with the latest results from the > &

Re: NoSQL database agnostic persistence

Just a heads up, I updated the report with the latest results from the persistence work, as well as the tarball with raw results. -- Pierre Laporte @pingtimeout <https://twitter.com/pingtimeout> pie...@pingtimeout.fr On Wed, Mar 19, 2025 at 3:20 PM Pierre Laporte wrote: > Hi, > &

Polaris benchmarks proposal

benchmark, an HTML page is generated with interactive charts showing query performance over time, response time percentiles, etc... I would love to head your feedback on it. Pierre [1] https://github.com/apache/polaris/pull/1208 -- Pierre Laporte @pingtimeout <https://twitter.com/pingtime

Re: NoSQL database agnostic persistence

://github.com/apache/polaris/pull/1189 -- Pierre Laporte @pingtimeout <https://twitter.com/pingtimeout> pie...@pingtimeout.fr http://www.pingtimeout.fr/ On Mon, Mar 17, 2025 at 3:46 PM Jean-Baptiste Onofré wrote: > Hi Robert, > > Thanks for the update and the draft PR ! > > I w