Re: [DISCUSSION] Apache Polaris - Zulip Chat Bylaws

2024-09-03 Thread Dmitri Bourlatchkov
Thanks for writing these guidelines, Robert! They look pretty reasonable to me. Cheers, Dmitri. On Mon, Sep 2, 2024 at 5:06 AM Robert Stupp wrote: > Hi all, > > Since our "official" chat is on Zulip [1], we should have some rules for > it as well, so I started a document that I would like to sh

Re: Welcome to Apache Polaris (incubating) community

2024-09-05 Thread Dmitri Bourlatchkov
Nice site :) Thanks, JB! On Mon, Sep 2, 2024 at 2:59 AM Jean-Baptiste Onofré wrote: > Hi folks, > > As said in my previous email, the Apache Polaris website has been > published to https://polaris.apache.org. > It's a first version, it will be updated soon. > > Regards > JB > > On Thu, Aug 29, 2

Re: [VOTE] Apache Polaris - Zulip Project Chat Bylaws

2024-09-06 Thread Dmitri Bourlatchkov
+1 (nb) Cheers, Dmitri. On Fri, Sep 6, 2024 at 5:47 AM Robert Stupp wrote: > Hi all, > > Although there seems to be consensus on the proposal, I think it is good > to have a formal vote, since this is a bylaws document. > > I've opened a PR [1] to add the Chat Bylaws as a markdown document to >

Re: [DISCUSS] Supporting region attribute in vended credentials for AWS

2024-09-12 Thread Dmitri Bourlatchkov
On Thu, Sep 12, 2024 at 8:38 AM Robert Stupp wrote: > What would be the difference of the region specified in/for the > credentials and the region that's configured for the S3FileIO (via > LoadTableResult)? > > I think it may be a part of a larger discussion about table files (potentially) spread

Re: Polaris Community Virtual Sync

2024-10-11 Thread Dmitri Bourlatchkov
Thanks for organising this, Russell! Thursday 9 AM Pacific works for me. Cheers, Dmitri. On Fri, Oct 11, 2024 at 4:37 PM Russell Spitzer wrote: > Hi Y'all! > > I love mailing lists but sometimes I feel like it's good to have a nice > video conference to put faces to names and discuss issues sy

Re: Heading to Polaris 0.9 release and beyond

2024-10-18 Thread Dmitri Bourlatchkov
+1 to "fast" major version. Cheers, Dmitri. On Fri, Oct 18, 2024 at 5:17 AM Jean-Baptiste Onofré wrote: > Hi folks, > > First of all, thanks everyone for the first community meeting > yesterday. It was a great meeting, very interesting discussions. > You can find the meeting notes on the Polari

Re: Heading to Polaris 0.9 release and beyond

2024-10-21 Thread Dmitri Bourlatchkov
to manage > production impact? While we can't completely avoid user forks, we want to > minimize divergence. > > Yufei > > > On Fri, Oct 18, 2024 at 6:33 AM Dmitri Bourlatchkov > wrote: > > > +1 to "fast" major version. > > > > Ch

Re: [DISCUSS] Interoperability definition

2024-10-25 Thread Dmitri Bourlatchkov
x27;s worth adding a section in the documentation to define > "Polaris interoperability". > > Regards > JB > > On Wed, Oct 23, 2024 at 11:23 PM Dmitri Bourlatchkov > wrote: > > > > Hi All, > > > > The Polaris GH repo has this statement:

[DISCUSS] Persistence: conditional SQL UPDATE

2024-10-31 Thread Dmitri Bourlatchkov
Hi All, As discussed in the sync call today, I'm opening this email thread with the aim to clarify our approach to Persistence. So this is specific to storing data in RDBMS (leaving NoSQL to another thread). IIRC, it was mentioned in the sync call that statements like "UPDATE ... WHERE key = X a

[DISCUSS] Interoperability definition

2024-10-23 Thread Dmitri Bourlatchkov
Hi All, The Polaris GH repo has this statement: "Apache Polaris, the interoperable, open source catalog for Apache Iceberg". How is this "interoperability" defined? Do we have any docs that talk about that? If not, what is the general perception in this community about what is considered interop

Re: [DISCUSS] Apache Polaris powered by Quarkus

2024-11-13 Thread Dmitri Bourlatchkov
2. Refactor polaris-service into 2 modules: e.g. "polaris-service-common" and "polaris-service-dropwizard" – I believe Dmitri is already working on this Confirming: I do have some WIP for this. On Wed, Nov 13, 2024 at 2:13 PM Alex Dutra wrote: > Hi Michael, hi all, > > I had a look at

Re: [DISCUSS] Apache Polaris powered by Quarkus

2024-11-13 Thread Dmitri Bourlatchkov
Re: test classes - I'd propose to make a new module, e.g. `polaris-test` to contain abstract (actually or in spirit) test code, which would then be picked up by DW and Quarkus-specific actual test classes. I'm sure we can isolate env. details in JUnit5 extensions and inject relevant access objects

Re: [PROPOSAL] Thinking about first Apache Polaris release

2024-09-23 Thread Dmitri Bourlatchkov
>From my POV, I'd propose to resolve the OAuth token endpoint concern [1] before the initial release. I guess it might be a rather big refactoring, but this issue is already generally accepted as a security concern in the Iceberg community, so I think it would be preferable to resolve it before th

Re: [PROPOSAL] Thinking about first Apache Polaris release

2024-09-23 Thread Dmitri Bourlatchkov
, 2024 at 1:50 PM Jean-Baptiste Onofré > > wrote: > > > > > Hi Dmitri > > > > > > It makes sense to me. > > > > > > Regards > > > JB > > > > > > Le lun. 23 sept. 2024 à 20:01, Dmitri Bourlatchkov > > > a écrit : &

Re: [PROPOSAL] Thinking about first Apache Polaris release

2024-09-23 Thread Dmitri Bourlatchkov
ust > start at 1. > > On Mon, Sep 23, 2024 at 1:50 PM Jean-Baptiste Onofré > wrote: > > > Hi Dmitri > > > > It makes sense to me. > > > > Regards > > JB > > > > Le lun. 23 sept. 2024 à 20:01, Dmitri Bourlatchkov > > a écrit

Re: [PROPOSAL] Thinking about first Apache Polaris release

2024-09-23 Thread Dmitri Bourlatchkov
s OAuth2 Flow IIRC. Cheers, Dmitri. On Mon, Sep 23, 2024 at 3:23 PM Dmitri Bourlatchkov < dmitri.bourlatch...@dremio.com> wrote: > > If we remove the endpoint in Polaris, clients prior to that release will > have no mechanism for generating a token. > > The basic problem is that

Re: [PROPOSAL] Thinking about first Apache Polaris release

2024-09-24 Thread Dmitri Bourlatchkov
w and > 2.0 > > > if Polaris doesn't have it. > > > 3. Support for third-party token endpoints was introduced about seven > > > months ago in both Java and Python, which isn't too far back, as > Michael > > > pointed out. > > > > > > WDYT?

Re: [ANNOUNCE] Welcoming new committers

2024-09-26 Thread Dmitri Bourlatchkov
Congrats everyone :) On Thu, Sep 26, 2024 at 8:41 AM Jean-Baptiste Onofré wrote: > Hi folks, > > We are very happy to announce new committers to the Apache Polaris > (incubating) podling ! > > * Anna Filippova > * Eric Maynard > * Michael Collado > * Yufei Gu > * Yuya Ebihara > > Thanks for all

Re: [PROPOSAL] Apache Polaris "meetup" during CommunityOverCode and start the community meetings

2024-09-20 Thread Dmitri Bourlatchkov
+1 to community meetings every two weeks. Cheers, Dmitri. On Fri, Sep 20, 2024 at 11:09 AM Jean-Baptiste Onofré wrote: > Hi folks, > > I'm happy to share that we will have a talk about Apache Polaris > (incubating) during CommunityOverCode NA on Monday, October 7th at > 4:50pm (https://communit

Re: [ANNOUNCE] Welcome to new Polaris committers !

2024-11-06 Thread Dmitri Bourlatchkov
Congratulations, Alex and Yong! On Wed, Nov 6, 2024 at 3:57 AM Jean-Baptiste Onofré wrote: > Hi folks > > We are pleased to announce new committers on the Apache Polaris > (incubating) podling: > > - Alex Dutra > - Yong Zheng > > Welcome and congrats ! Thanks for your contribution on Polaris ! >

Re: [VOTE] Release Apache Polaris 0.9.0-incubating

2024-11-20 Thread Dmitri Bourlatchkov
t; > > Good catch. > > > > I think it's severe enough to cancel rc1 and prepare a new one > > including eclipselink fix. > > > > Thoughts ? > > > > Regards > > JB > > > > On Wed, Nov 20, 2024 at 9:21 PM Dmitri Bourlatchkov >

Re: [VOTE] Release Apache Polaris 0.9.0-incubating

2024-11-20 Thread Dmitri Bourlatchkov
I've just noticed that RC1 includes https://github.com/apache/polaris/pull/438, which, I think, makes bootstrapping on EclispeLink effectively impossible, because there's no user-level way to discover the generated root secret. >From my personal point of view it is not a release blocker, so I'm ke

Re: [DISCUSS] Apache Polaris powered by Quarkus

2024-11-27 Thread Dmitri Bourlatchkov
I opened the other day. Take a look at this one > and let me know what you think. > > On Tue, Nov 26, 2024 at 11:50 AM Dmitri Bourlatchkov > wrote: > > > Hi Mike, > > > > I had a brief, but end-to-end look at your latest HK2 branch. Overall, I > > think

Re: [DISCUSS] Apache Polaris powered by Quarkus

2024-11-27 Thread Dmitri Bourlatchkov
er day. Take a look at this one > and let me know what you think. > > On Tue, Nov 26, 2024 at 11:50 AM Dmitri Bourlatchkov > wrote: > > > Hi Mike, > > > > I had a brief, but end-to-end look at your latest HK2 branch. Overall, I > > think it is a good refactori

Re: [DISCUSS] Apache Polaris powered by Quarkus

2024-11-27 Thread Dmitri Bourlatchkov
Sorry, that one is for Web Apps On Wed, Nov 27, 2024 at 5:07 PM Dmitri Bourlatchkov < dmitri.bourlatch...@dremio.com> wrote: > I see jakarta.ws.rs.Produces in jakarta.ws.rs-api 3.0.0... Does HK2 have a > lower version of jakarta? > > On Wed, Nov 27, 2024 at 4:22 PM Michae

Re: [VOTE] Release Apache Polaris 0.9.0-incubating

2024-11-18 Thread Dmitri Bourlatchkov
Hi JB, I can see only your key (one) in the KEYS file... Is that normal? Thanks, Dmitri. On Sun, Nov 17, 2024 at 1:00 AM Jean-Baptiste Onofré wrote: > Hi everyone, > > I propose that we release the following RC as the official Apache > Polaris 0.9.0-incubating release. > > * This corresponds t

Re: [VOTE] Release Apache Polaris 0.9.0-incubating

2024-11-18 Thread Dmitri Bourlatchkov
I noticed that the apache-polaris-0.9.0-incubating.sha512 file contains the right hash, but it's format does not match the output of `sha512sum apache-polaris-0.9.0-incubating.tar.gz` -- files names are not listed after hashes... As a result, sha512sum -c apache-polaris-0.9.0-incubating.sha512 ret

Re: [VOTE] Release Apache Polaris 0.9.0-incubating

2024-11-18 Thread Dmitri Bourlatchkov
tc). I will create the issues/PRs to fix that with > the release process documentation. > > Regards > JB > > On Mon, Nov 18, 2024 at 3:03 PM Dmitri Bourlatchkov > wrote: > > > > I noticed that the apache-polaris-0.9.0-incubating.sha512 file contains > the > &

Re: [DISCUSS] Apache Polaris powered by Quarkus

2024-11-19 Thread Dmitri Bourlatchkov
I had a quite brief look at Mike's HK2 branch, so apologies if I'm missing the bigger picture. Still, I see that MetaStoreManagerFactory no longer extends the DW Discoverable, but it extends the HK2 Factory class now. So we're basically trading one framework for another as a code-level runtime dep

Re: [DISCUSS] Apache Polaris powered by Quarkus

2024-11-26 Thread Dmitri Bourlatchkov
dependencies on the specific realm-scoped beans they need and let > > the CDI framework work out where the beans come from. This is basically > > what I was doing with the MetaStoreManagerFactory interface - as I said, > I > > was just being lazy by extending Factory directly. > &

Re: [DISCUSS] Apache Polaris powered by Quarkus

2024-11-13 Thread Dmitri Bourlatchkov
To avoid confusion: I mean `polaris-test` is for the integration kind of tests that require some support from the environment, like starting a Polaris server. On Wed, Nov 13, 2024 at 4:27 PM Dmitri Bourlatchkov < dmitri.bourlatch...@dremio.com> wrote: > Re: test classes - I'd pro

Re: [DISCUSS] Apache Polaris powered by Quarkus

2024-11-13 Thread Dmitri Bourlatchkov
Wed, Nov 13, 2024 at 4:28 PM Dmitri Bourlatchkov < dmitri.bourlatch...@dremio.com> wrote: > To avoid confusion: I mean `polaris-test` is for the integration kind of > tests that require some support from the environment, like starting a > Polaris server. > > On Wed, Nov 13, 2

[DISCUSS] Polaris Persistence Contract

2024-12-03 Thread Dmitri Bourlatchkov
Hi All, I believe it was already discussed elsewhere that it is valuable to allow Apache Polaris to be extensible, and in particular extensible in how it interacts with its own Persistence backend (not to be confused with Iceberg data storage). I’d like to formalize the expectations Polaris Core

Re: [DISCUSS] Vending raw credentials in Polaris

2024-12-09 Thread Dmitri Bourlatchkov
Looking a bit more into the PR, I think it is primarily about avoiding an STS call rather than about "raw" credentials. I think the STS requirement can, indeed, be a blocker for some custom S3 implementations. If we want to support those, we could allow the admin user to configure a separate set

Re: [DISCUSS] Vending raw credentials in Polaris

2024-12-09 Thread Dmitri Bourlatchkov
(note: I did not review the PR) On-prem systems usually have different security perimeters than cloud systems. While vending long-term credentials by default is too risky, I agree, why would Polaris restrict that in controlled environments where the admin user explicitly wants to enable that (e.g

Re: [DISCUSS] Vending raw credentials in Polaris

2025-01-06 Thread Dmitri Bourlatchkov
On Mon, Jan 6, 2025 at 2:12 PM Eric Maynard wrote: > > why would Polaris restrict that in controlled environments > > To Michael's point, I think this kind of reasoning is a little dangerous. > We need to clearly define what Polaris will and won't support, rather than > adopting the mentality tha

[DISCUSS] TestEnvironmentExtension

2025-01-06 Thread Dmitri Bourlatchkov
Hi All, I'd like to clarify the role and intended use of TestEnvironmentExtension in the Polaris test codebase. The OSS part of the code that relates to TestEnvironmentExtension does not appear to demonstrate how non-dropwizard environments function. I'd appreciate it if people could share their

Re: Welcome new committer

2025-01-07 Thread Dmitri Bourlatchkov
7, 2025 at 7:33 PM Alex Dutra > > > > > > > > > wrote: > > > > > > > > > Hi all, > > > > > > > > > > That's great news, congratulations Dmitri and welcome to the team! > > > > > > > > > > Thanks,

Re: [DISCUSS] Polaris Persistence architecture, refactor, support for new backends (including MongoDB, DynamoDB)

2025-02-04 Thread Dmitri Bourlatchkov
Nice document, Dennis! Thanks for putting it together. I find it very useful. I added a lot of comments, not as critique, but hoping to achieve clarity. One thing which is missing from the doc, IMHO, is the discussion of handling concurrent reads / writes made by different instances of the Polar

Re: [VOTE] Release Apache Polaris 0.9.0-incubating (rc5)

2025-02-07 Thread Dmitri Bourlatchkov
+1 (nb) Verified: * Checksum * Signatures * ./gradlew clean build Cheers, Dmitri. On Thu, Feb 6, 2025 at 11:55 AM Jean-Baptiste Onofré wrote: > Hi folks, > > We did new fixes on NOTICE and LICENSE fix in the source distribution. > > This is the vote for Apache Polaris 0.9.0-incubating rc5. > >

Re: [DISCUSS] Polaris Persistence architecture, refactor, support for new backends (including MongoDB, DynamoDB)

2025-02-05 Thread Dmitri Bourlatchkov
Hi JB, I think your doc also has a lot of valuable information and proposals. Unfortunately, I'm afraid my point about lack of concurrent updates discussion applies to your doc too. I think it is important to define conflicting change resolution at the persistence layer across all backends becau

Re: [DISCUSS] Polaris Persistence architecture, refactor, support for new backends (including MongoDB, DynamoDB)

2025-02-05 Thread Dmitri Bourlatchkov
> > 1. Secondary indexes make lookups by-id and by-name "atomic" in the face of > concurrent updates. Some databases that support transactions but not > secondary indexes will use transactions over "index tables" I propose to delegate this to the Persistence implementation and not expose anything

Re: Podling Polaris Report Reminder - February 2025

2025-02-05 Thread Dmitri Bourlatchkov
LGTM. Thanks, JB! Cheers, Dmitri. On Wed, Feb 5, 2025 at 11:58 AM Jean-Baptiste Onofré wrote: > Hi folks, > > I drafted a first incubator report: > https://cwiki.apache.org/confluence/display/INCUBATOR/February2025#polaris > > Can you please take a look and let me know if you wanna add content

Re: [DISCUSS] Polaris Persistence architecture, refactor, support for new backends (including MongoDB, DynamoDB)

2025-02-06 Thread Dmitri Bourlatchkov
Thanks all for the very informative discussion on this topic in the community sync meeting today! Let's have a dedicated discussion session as suggested. Dennis: As to your doc, would you mind isolating caching concerns into a separate section so that we could think about the main Persistence API

[DISCUSS] Separate Helm charts release

2025-01-30 Thread Dmitri Bourlatchkov
Hi All, PR 912 [1] prompted this discussion. I believe it is valuable to release helm charts separately from the main source / binary bundle primarily because the lifecycle of the charts is different from java code and often requires changes that are not connected to service implementations. I'd

Re: Zulip & Slack

2025-01-31 Thread Dmitri Bourlatchkov
Good points from JB about compliance with community guidelines! Re: Zulip in general. I've had a lot of user conversations in Nessie via Zulip lately, and I can say that Zulip is better organised in that regard. Its topics are more like email threads, which IMHO maps better to `dev` ML discussions

Re: [DISCUSS] Polaris Federated Principals and Roles

2024-12-17 Thread Dmitri Bourlatchkov
Hi Mike, I left some comments in the doc, but overall it looks good to me :) I still think there are some hidden dependencies on Persistence. For example, whether and how we can have composite keys for persisted federated entities... but I guess we can work that out later. Also, I think it is im

Re: S3 Tables and Apache Polaris Integration.

2024-12-18 Thread Dmitri Bourlatchkov
Hi All, This looks like a valuable feature for users. I wonder about multi-table commits, though. These commits are supposed to be atomic in the Iceberg REST API, but I do not think a simple proxy for the S3 table API can ensure atomicity. Does S3 tables API support multi-table changes (I did no

Re: [PROPOSAL] Decomposition of PolarisMetaStoreManager interface

2024-12-13 Thread Dmitri Bourlatchkov
Thanks for this proposal, Mike! I made some comments on the doc. Would you mind adding a diagram with CDI scopes and interface instances (i.e. whether a particular object implements one or more interfaces)? As for me personally, it would be very helpful for understanding runtime implications bett

Re: Switch to Slack for Polaris Communication

2024-12-13 Thread Dmitri Bourlatchkov
Apparently images are discarded... Here's the text that was my concern: "Don’t have an *@apache.org <http://apache.org>* email address? Contact the workspace administrator at *Apache-Polaris* for an invitation." Cheers, Dmitri. On Fri, Dec 13, 2024 at 12:51 P

Re: Switch to Slack for Polaris Communication

2024-12-13 Thread Dmitri Bourlatchkov
apache > email address. Many people joined with their personal email addresses. It > may prompt with an apache email, but any email should be fine. > > Yufei > > > On Fri, Dec 13, 2024 at 8:21 AM Dmitri Bourlatchkov > wrote: > > > It looks like the same Apache email ob

[DISCUSS] Choosing backend at runtime

2024-12-13 Thread Dmitri Bourlatchkov
Hi All, Currently using Polaris with a particular backend apparently requires special build-time options. For example, targeting EclipseLink with PostgreSQL needs "-PeclipseLink=true -PeclipseLinkDeps=org.postgresql:postgresql:42.7.4" at build (!) time [1]. This email is to discuss our approach

Re: Switch to Slack for Polaris Communication

2024-12-13 Thread Dmitri Bourlatchkov
t; [1] > > nor > > > > in a > > > > > > > > > related discussion & vote [2]. I don’t think that it’s a > good > > > > idea to > > > > > > > > > change the project’s official chat after just a couple of > > weeks, > &g

Re: Switch to Slack for Polaris Communication

2024-12-13 Thread Dmitri Bourlatchkov
Thanks, Yufei! The new link works well :) On Fri, Dec 13, 2024 at 1:23 PM Yufei Gu wrote: > The invitation link( > > https://join.slack.com/t/apache-polaris/shared_invite/zt-2w1fddyh3-zqCeeJwn7wNvhn3mVT5njQ > ) > works well. The Iceberg site also uses an invitation link. Updated the site > in ht

[DISCUSS] Incubating version number for snapshots

2024-11-21 Thread Dmitri Bourlatchkov
Hi All, This may be nitpicky but the current version in local builds is "1.0.0-incubating-SNAPSHOT", which does not appear to follow semver [1]. I suppose a semver-compatible version number would be "1.0.0-incubating.SNAPSHOT" (a dot instead of the second dash). Would it work from your POV? Tha

Re: [DISCUSS] Incubating version number for snapshots

2024-11-21 Thread Dmitri Bourlatchkov
-y-z.--. > > Yufei > > > On Thu, Nov 21, 2024 at 11:30 AM Laurent Goujon > > wrote: > > > Wouldn't that cause issues with maven repositories when trying to upload > > the artifacts (possibly miscategorized as release artifacts instead of > &g

Re: Switch to Slack for Polaris Communication

2024-12-06 Thread Dmitri Bourlatchkov
of the mailing list or GitHub and I’m already in slack all > day every day anyway and given that Zulip is _only_ ever used for > occasional user questions… > > I like open source projects, but… I can’t do it all. > > Mike > > On Fri, Dec 6, 2024 at 11:56 AM Dmitri Bour

Re: Unstructure data support in Apache Polaris

2024-12-06 Thread Dmitri Bourlatchkov
Hi Yufei, Interesting proposal. I commented in the doc. WDYT about using Iceberg metadata to list the stage files (in manifests)? Thanks, Dmitri. On Thu, Dec 5, 2024 at 6:21 PM Yufei Gu wrote: > Hi Folks, > > Polaris has become a cornerstone for managing structured data across > diverse proce

Re: [DISCUSS] Polaris Persistence Contract

2024-12-06 Thread Dmitri Bourlatchkov
gt; wrote: > > > I think this is a great idea. Even if we put aside the NoSQL / RDBMS > point, > > simply clarifying the roles & responsibilities of the persistence > > interface(s) would be a welcome improvement. > > > > --EM > > > > On Tue, Dec 3, 2024 at

Re: Switch to Slack for Polaris Communication

2024-12-06 Thread Dmitri Bourlatchkov
Zulip is an OSS tool, at least it has a functional OSS server [1] and client [2] variants (Apache-licensed). Apache Polaris being an OSS project itself, I think it would be nice to support other OSS projects like Zulip by expanding user awareness by using it for developer chats and user support.

Re: Switch to Slack for Polaris Communication

2024-12-06 Thread Dmitri Bourlatchkov
> 2. *Performance and Stability*: I personally use Zulip Desktop on Linux and the Zulip App on my phone and do not recall any serious performance and stability problems. There was one event a long time ago, when a new desktop client version became unusable, but it was solved quickly with a patch

Re: [VOTE] Release Apache Polaris 0.9.0-incubating (rc2)

2025-01-10 Thread Dmitri Bourlatchkov
+1 (nb) Verified signature, checksum. JB: I believe you mentioned in the community sync call that you were going to share some info on how releases are supposed to be verified :) Cheers, Dmitri. On Wed, Jan 8, 2025 at 11:01 AM Jean-Baptiste Onofré wrote: > Hi folks, > > As mentioned in anothe

[DISCUSS] Special Storage settings in CI

2025-01-10 Thread Dmitri Bourlatchkov
Hi All, I see that some tests (e.g. for GCP and Azure) apparently require special settings to be able to run. Would there be any objections to gradually converting those tests to use emulators or containerized storage (like MinIO) and hence have no special settings or secrets? Thanks, Dmitri.

Re: [DISCUSS] Special Storage settings in CI

2025-01-10 Thread Dmitri Bourlatchkov
ts less often. For > example, only prior to merging or to cutting a release. > > --EM > > On Fri, Jan 10, 2025 at 3:36 PM Dmitri Bourlatchkov > wrote: > > > Hi All, > > > > I see that some tests (e.g. for GCP and Azure) apparently require special > > se

[DISCUSS] Quarkus-based docker images (#610)

2025-01-22 Thread Dmitri Bourlatchkov
Hi All, For visibility: PR#610 (docker changes) [1] has some "request changes" comments. It looks like they got addressed. If there are still concerns, please comment again. Thanks, Dmitri. [1] https://github.com/apache/polaris/pull/610

Re: [VOTE] Release Apache Polaris 0.9.0-incubating (rc3)

2025-01-21 Thread Dmitri Bourlatchkov
/apache/polaris/version/TestPolarisVersion.java:113: > error: illegal start of expression > } > ^ > > > Patch attached > > On Tue, Jan 21, 2025 at 1:19 PM Dmitri Bourlatchkov > wrote: > >> +1 (nb) >> >> Validated checksum, signature and local build

[DISCUSS] Bootstrap env. vars simplification (#806)

2025-01-21 Thread Dmitri Bourlatchkov
Hi All, For the sake of visibility: PR #806 [1] proposes a change to the bootstrap environment variable value format. I personally think it is a net usability improvement, so I approved it in GH. If you have a preference or opinion please review and comment/approve on the PR. Thanks, Dmitri. [1

Re: [PROPOSAL] Improve our "collaboration schema"

2025-01-22 Thread Dmitri Bourlatchkov
On Wed, Jan 22, 2025 at 12:58 PM Jean-Baptiste Onofré wrote: > It's not a problem to have several people working on the same feature. > > By duplicate PRs, I mean that you open a PR, where you get comments, > changes requested, etc. Then, you close this PR to open a new one (on > the same topic).

Re: [PROPOSAL] Improve our "collaboration schema"

2025-01-22 Thread Dmitri Bourlatchkov
Hi JB, What do you mean by "No duplicated PRs is allowed (in order to keep history and comments)"? Maybe a terminology gap on my part :) but what is a "duplicated PR"? Do you mean multiple people working on the same feature? Or people submitting multiple PRs with alternative implementations? Ho

Re: [DISCUSS] @RolesAllowed in generated REST API classes

2025-01-24 Thread Dmitri Bourlatchkov
https://github.com/apache/polaris/pull/875 On Fri, Jan 24, 2025 at 11:02 AM Dmitri Bourlatchkov wrote: > Thanks for the context, Mike! I'll open a PR to remove them. > > Cheers, > Dmitri. > > On Fri, Jan 24, 2025 at 10:46 AM Michael Collado > wrote: > >> They

Re: Polaris has two websites

2025-01-23 Thread Dmitri Bourlatchkov
I think redirection (simply to the home page) is preferable, if possible. Cheers, Dmitri. On Thu, Jan 23, 2025 at 2:35 PM Jack Ye wrote: > +1. We can redirect it to polaris.apache.org if we want to redirect > existing users. > > -Jack Ye > > On Thu, Jan 23, 2025 at 10:27 AM Yufei Gu wrote: > >

Re: [VOTE] Release Apache Polaris 0.9.0-incubating (rc4)

2025-01-23 Thread Dmitri Bourlatchkov
+1 (ns) Verified: * Checksum * Signarures * ./gradlew clean assemble * ./gradlew test * Start with in-memory MetaStore * Create catalog via ./polaris CLI with FILE storage * Basic create table / select in Spark Cheers, Dmitri. On Wed, Jan 22, 2025 at 3:51 AM Jean-Baptiste Onofré wrote: > Hi f

Re: [PROPOSAL] Improve our "collaboration schema"

2025-01-24 Thread Dmitri Bourlatchkov
On Thu, Jan 23, 2025 at 5:15 AM Jean-Baptiste Onofré wrote: > Let's rename from "guidelines" to "good practices & advices" :) > +1 to that. I'm not sure it worth trying to "codify a spirit" (from previous emails), but I think having good advice is helpful to instill the spirit of goodwill and co

Re: [DISCUSS] @RolesAllowed in generated REST API classes

2025-01-24 Thread Dmitri Bourlatchkov
enough for Polaris. > > > > Thanks, > > Alex > > > > On Fri, Jan 24, 2025 at 3:25 PM Dmitri Bourlatchkov > > wrote: > > > > > Hi All, > > > > > > Currently the code generated for various REST API endpoints contains > > > &qu

[DISCUSS] @RolesAllowed in generated REST API classes

2025-01-24 Thread Dmitri Bourlatchkov
Hi All, Currently the code generated for various REST API endpoints contains "@RolesAllowed" annotations. Do people find them critical? >From my POV, it is preferable to delegate all authorization logic to PolarisAuthorizer implementations and remove any framework-specific ways to control access

Re: Proposal: Use of Realm Instead of RealmId

2025-01-27 Thread Dmitri Bourlatchkov
Thanks for starting this discussion, Yufei! I think it is important to clarify this sooner rather than later. >From my POV option 2 (String) is cumbersome. The biggest reasons being more complex CDI injection (as already noted by other people) but also more complicated usage searches in IDEs. As

Re: Proposal: Use of Realm Instead of RealmId

2025-01-27 Thread Dmitri Bourlatchkov
roxy to the whole entity directly, that's imho > easier. > > Thanks, > > Alex > > On Mon, Jan 27, 2025 at 4:05 PM Dmitri Bourlatchkov > wrote: > > > Thanks for starting this discussion, Yufei! I think it is important to > > clarify this sooner rather than la

Re: [Proposal] Distinguish Iceberg vs non-Iceberg APIs in Polaris

2025-01-28 Thread Dmitri Bourlatchkov
Using multiple YAML files per API / API area sounds good to me. We should definitely use the Iceberg REST API YAML without any modifications to avoid accidental mistakes. Cheers, Dmitri. On Tue, Jan 28, 2025 at 2:28 AM Honah J. wrote: > Hi everyone, > > Thanks for the proposal and all the grea

Re: [VOTE] Release Apache Polaris 0.9.0-incubating (rc3)

2025-01-21 Thread Dmitri Bourlatchkov
+1 (nb) Validated checksum, signature and local build with JDK 21. Ran and smoke-tested by creating a catalog via ./polaris CLI. Note: I did observe some WARN log messages from HK2, but I do not think those are blockers for this release. Cheers, Dmitri. On Mon, Jan 20, 2025 at 2:32 AM Jean-Bapt

Re: Welcome Dennis Huo as new committer

2025-01-13 Thread Dmitri Bourlatchkov
Welcome Dennis! Cheers, Dmitri. On Mon, Jan 13, 2025 at 2:38 AM Jean-Baptiste Onofré wrote: > Hi folks, > > > We are very happy to announce Dennis Huo as new Apache Polaris > (incubating) podling committer ! > > Welcome Dennis and congrats ! Thanks for all your work on Polaris ! > > Regards > J

[DISCUSS] Drop In-memory MetaStore in favour of H2

2025-01-14 Thread Dmitri Bourlatchkov
Hi All, Given that it is possible to run EclipseLink with H2 in memory, is there value in keeping a separate in-memory MetaStore implementation? My main concern is that the plain in-memory MetaStore is significantly different from the EclipseLink metastore and might deviate in behaviour. With tha

Re: [DISCUSS] Drop In-memory MetaStore in favour of H2

2025-01-14 Thread Dmitri Bourlatchkov
ly using JDBC (without EclipseLink) ? > > Quarkus Panache is not really an option for now due to license issue > > (Hibernate ORM). > > > > Regards > > JB > > > > On Tue, Jan 14, 2025 at 4:12 PM Dmitri Bourlatchkov > > wrote: > > > > > > Hi

Re: Switch to Quarkus – when is the right time?

2024-12-31 Thread Dmitri Bourlatchkov
Hi All, To clarify my intent behind #590 [1], the main idea behind this refactoring is to establish something akin to a TCK [2] for Polaris. These tests are meant to be used to validate that an alternative server runtime (e.g. Quarkus) behaves according to what is expected from the Polaris Server

[DISCUSS] Clarification of Persistence SPI

2025-01-09 Thread Dmitri Bourlatchkov
Hi All, I have been experimenting with running Polaris on PostgreSQL where multiple servers access the same database. I present two specific test cases in the appendixes below for review. The test cases are possible in real life even though they may look a bit contrived in the current state of t

Re: [DISCUSS] Preparing 0.10.0 release including binary distributions

2025-03-19 Thread Dmitri Bourlatchkov
Verify all legal aspects and the release is OK for the IPMC > > > So, 0.10.0 or 1.0.0-preview work. Personally, I consider 1.0.0-preview > > > more meaningful because, without considering the scope/content, it's > > > really what it is: a preview release to test our

Re: Catalog Federation API Spec updates - adding ConnectionConfigInfo

2025-03-19 Thread Dmitri Bourlatchkov
Thanks, Dennis, for starting this discussion. As for me the PR looks ok in its current state, although I posted a few comments. I tend to think that in order to proceed with the federated catalogs implementation, it is probably best to merge the PR now and later rework the parts that deal with ex

Re: Catalog Federation API Spec updates - adding ConnectionConfigInfo

2025-03-20 Thread Dmitri Bourlatchkov
h an internal SecretsManager interface to avoid storing > plaintext secrets directly within the CatalogEntity. > > > > > On Wed, Mar 19, 2025 at 3:53 PM Dmitri Bourlatchkov > wrote: > > > Thanks, Dennis, for starting this discussion. > > > > As for me t

Re: [DISCUSS] Preparing 0.10.0 release including binary distributions

2025-03-21 Thread Dmitri Bourlatchkov
w release > > > purpose: > > > - include binary distributions > > > - include legal aspect for all artifacts > > > - submit the release to the mentors and, if passed, to the IPMC > > > The idea is to validate a release with binary distributions/artif

Re: [DISCUSS] Preparing 0.10.0 release including binary distributions

2025-03-17 Thread Dmitri Bourlatchkov
ri, Mar 14, 2025 at 1:02 PM Jean-Baptiste Onofré < > j...@nanthrax.net> > > > > wrote: > > > > > > > > > Hi Dmitri > > > > > > > > > > > https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fhub.docker.com

Re: [DISCUSS] Preparing 0.10.0 release including binary distributions

2025-03-14 Thread Dmitri Bourlatchkov
As we have a Polaris repo on Docker HUB, I will also include docker image > check. > > Regards > JB > > On Fri, Mar 14, 2025 at 1:53 PM Dmitri Bourlatchkov > wrote: > > > > I think it's a good idea. Thanks for taking care of this, JB! > > > > What i

Re: [DISCUSS] Preparing 0.10.0 release including binary distributions

2025-03-14 Thread Dmitri Bourlatchkov
.0. > > Otherwise, either one is fine. > > > > Yufei > > > > > > On Fri, Mar 14, 2025 at 1:02 PM Jean-Baptiste Onofré > > wrote: > > > > > Hi Dmitri > > > > > > https://hub.docker.com/r/apache/polaris > > >

Re: [DISCUSS] Preparing 0.10.0 release including binary distributions

2025-03-14 Thread Dmitri Bourlatchkov
t; be published as well: > > > https://github.com/apache/polaris/blob/main/quarkus/admin/src/main/docker/Dockerfile.jvm > > Thanks, > > Alex > > > > On Fri, Mar 14, 2025 at 10:23 PM Dmitri Bourlatchkov > wrote: > > > I agree that a simple docker p

Re: [DISCUSS] Release cycle for Spark Client

2025-04-04 Thread Dmitri Bourlatchkov
Hi Yun, Your proposal LGTM. However, regarding compatibility, I think this information has to be tracked regardless of the release cycle, because users can mix different client / server versions in their environments. Cheers, Dmitri. On Tue, Mar 25, 2025 at 5:01 PM yun zou wrote: > Hi Team, >

Re: [VOTE] Add 'inherited' and 'namespaces' Fields to GetApplicablePolicies API Response

2025-04-04 Thread Dmitri Bourlatchkov
t > the field identifies a single namespace and differentiates it from the > namespace parameter, which is a single string. > > Please continue to vote. > > Thanks, > Honah(Jonas) > > On Fri, Apr 4, 2025 at 4:42 PM Dmitri Bourlatchkov > wrote: > > >

Re: [VOTE] REST API changes for Catalog Federation - addition of ConnnectionConfigInto to ExternalCatalog

2025-04-05 Thread Dmitri Bourlatchkov
+1 LGTM. Cheers, Dmitri. On Tue, Mar 25, 2025 at 12:30 AM Dennis Huo wrote: > Hello all, > > We've had some productive discussion in various places on the mailing list, > in the github PR, and in the live community sync now about the initial > minimal updates to the Polaris REST API (under the

Re: Polaris has two websites

2025-03-25 Thread Dmitri Bourlatchkov
Thanks, Anna! I see that https://polaris.io redirects to https://polaris.apache.org/ alright. However, http://polaris.io (no TLS) responds with a 500 error. It's a minor issue, but it would be nice to fix it :) Thanks, Dmitri. $ curl -v http://polaris.io/ * Trying 104.18.21.161:80... *

Re: [ANNOUNCE] Welcome Dmitri Bourlatchkov, Dennis Huo and Yufei Gu to the Apache Polaris PPMC

2025-03-27 Thread Dmitri Bourlatchkov
Thanks all for the warm welcome :) Cheers, Dmitri. On Wed, Mar 26, 2025 at 5:47 PM Russell Spitzer wrote: > Hi y'all! > > I'm excited to let the Polaris Community know that the PPMC has added three > new members. Dmitri Bourlatchkov, Dennis Huo and Yufei Gu are al

Re: [Discuss] Change EntityTable properties and internal properties from TEXT to JSONB

2025-04-01 Thread Dmitri Bourlatchkov
Using JSONB in the PostgreSQL-specific implementation sounds reasonable to me. My impression from the in-progress PRs on this subject is that the effort is focused on PostgreSQL and reusing the SQL schema in different RDBMS systems is not actually expected (or planned). On the topic of compatibil

Re: [DISCUSS] Preparing 0.10.0 release including binary distributions

2025-03-15 Thread Dmitri Bourlatchkov
I think it's a good idea. Thanks for taking care of this, JB! What is included in the binary distribution? Just jars or docker too? Side note: we should probably adjust PR #1070 [1] since the first release number is going to be different. [1] https://github.com/apache/polaris/pull/1170 Cheers,

Re: [DISCUSS] Rollback REPLACE commit on conflicts

2025-04-18 Thread Dmitri Bourlatchkov
Hi Prashant, Sorry for the delayed reply and apologies if I missed some relevant discussion. As I understand the catalog could remove snapshots that come in-between previous and current snapshots from the perspective of one of the clients. Can we be sure that the removed snapshot does not have

  1   2   3   4   >