Re: Clear separation of REST APIs

Thanks Robert for bringing this up! I see we mentioned two problems in this thread: 1) polaris-catalog-service.yaml contains reference to all catalog APIs including the ones for Iceberg and our Polaris specific APIs, which could be confusing to the users or new developers. 2) Polaris specific

Re: [ANNOUNCE] Welcome Dmitri Bourlatchkov, Dennis Huo and Yufei Gu to the Apache Polaris PPMC

Congratulations! Best Regards, Yun On Wed, Mar 26, 2025 at 4:28 PM Michael Collado wrote: > Wow! Awesome news. Congrats folks! > > Mike > > On Wed, Mar 26, 2025 at 2:47 PM Russell Spitzer > > wrote: > > > Hi y'all! > > > > I'm excited to let the Polaris Community know that the PPMC has added >

[DISCUSS] Initial location for Spark Client

Hi Team, Along with the support of Generic Table, we are also introducing a Polaris Spark client to interact with the new APIs. To start the work, I propose to put the client in the Polaris main repo for now. As we proceed, we can decide whether to move it to a different repo depending on the com

[DISCUSS] Release cycle for Spark Client

Hi Team, Given that we are now introducing Spark Client, one thing we need to decide is the release cycle for the Spark Plugin. I propose to bundle the client release with Polaris main release like Iceberg. In that way, users will be able to get the client support for new APIs in the same release

Re: [DISCUSS] Initial location for Spark Client

ersion). I > will check the dep and distributed artifacts to verify there is no “legal” > issue. > > Regards > JB > > > Le mar. 25 mars 2025 à 21:20, yun zou a > écrit : > > > Hi Team, > > > > Along with the support of Generic Table, we are also

Re: [DISCUSS] Release cycle for Spark Client

information has to be > tracked regardless of the release cycle, because users can mix different > client / server versions in their environments. > > Cheers, > Dmitri. > > On Tue, Mar 25, 2025 at 5:01 PM yun zou > wrote: > > > Hi Team, > > > > Given that

Re: [DISCUSS] Deprecate Eclipse Link and make Relational JDBC as default

+1 (non-binding) On Fri, May 2, 2025 at 3:21 PM Yufei Gu wrote: > +1 on deprecating the EclipseLink backend. > > Yufei > > > On Fri, May 2, 2025 at 3:00 PM Prashant Singh > wrote: > > > Hi all, > > > > I’d like to get your thoughts on deprecating EclipseLink and making JDBC > > the default for

Re: [Discuss] Add `location` to generic table spec

25 at 5:22 PM Dmitri Bourlatchkov wrote: > No worries about the name. It is a possible alternative spelling :) > > On Wed, May 7, 2025 at 8:04 PM yun zou wrote: > > > Hi Dmitri, > > > > Sorry, I accidentally typed your name wrong in the previous reply!

Generic Table Code Architecture Proposal

Hi Team, Thanks all for the feedback and help provided for the Generic Table feature. As a follow up, we made up a proposal about the high level code architecture for supporting Generic Table APIs inside Polaris. Here is a link to the doc: https://docs.google.com/document/d/1aG2yLykCgf4yCm3Ha26mp

Re: Endpoint prefix name for Generic Table Support in Apache Polaris

> > > > [1] Jakarta Servlet Spec 6.0 - > > > > > https://jakarta.ee/specifications/servlet/6.0/jakarta-servlet-spec-6.0#uri-path-canonicalization > > [2] > > > > > https://github.com/projectnessie/nessie/blob/fbbbf8ae7ac0d961997c685afd32d5dceed90efc/api/

Generic Table Support in Apache Polaris

Apache Polaris currently only functions as a catalog for Apache Iceberg

Generic Table Support in Apache Polaris

Apache Polaris currently only functions as a catalog for Apache Iceberg

Updated invitation: Review: Generic Table Support in Apache Polaris @ Mon Feb 24, 2025 9am - 10am (PST) (dev@polaris.apache.org)

:20250222T013242Z ORGANIZER;CN=yun zou:mailto:yunzou.colost...@gmail.com UID:4n4nh6rcecni60d9umcrdan...@google.com ATTENDEE;CUTYPE=INDIVIDUAL;ROLE=REQ-PARTICIPANT;PARTSTAT=ACCEPTED;RSVP=TRUE ;CN=yun zou;X-NUM-GUESTS=0:mailto:yunzou.colost...@gmail.com ATTENDEE;CUTYPE=INDIVIDUAL;ROLE=REQ-PARTICIPANT

Endpoint prefix name for Generic Table Support in Apache Polaris

Hi All, Thanks all for attending the review today and providing your valuable feedback, I really appreciate it ! And apologize for the late notice for the review meeting here. I think we had a good discussion, and agreed to 1) Move on to provide a new Spark Catalog Plugin and new set of APIs with

Re: Endpoint prefix name for Generic Table Support in Apache Polaris

urrent proposal, but it will > look awkward if we have to add typed / specialized endpoints later (so > those API extensions may need yet another prefix). > * /catalog/generic-table/v1 - same as above > * /catalog/storage-entity/v1 > > Cheers, > Dmitri. > > On Mon, F

Re: [VOTE] REST API changes for Catalog Federation - addition of ConnnectionConfigInto to ExternalCatalog

+1 Best, Yun On Tue, Mar 25, 2025 at 9:22 AM Honah J. wrote: > +1 (non-binding) > > Best, > Honah (Jonas) > > On Tue, Mar 25, 2025 at 8:22 AM Yufei Gu wrote: > > > +1 > > > > Yufei > > > > > > On Tue, Mar 25, 2025 at 8:03 AM Prashant Singh > > wrote: > > > > > +1 (non-binding) > > > > > > Bes

Re: Polaris-XTable Proposal

Hi Rahil, Thanks a lot for the proposal! It is interesting. As Eric mentioned, we were also looking into this before, but at that time there wasn't Generic Table support yet. Now with the coming support for Generic Tables, other than the conversion performance, we might also want to look at more

Re: [Discuss] Add `location` to generic table spec

f the value of the new location attribute. > > > > > > - Is is one value or many? > > > - Is it a URI? > > > - Does it point to any particular file? > > > - Is it a common prefix of all files within a table? > > > - What happens when a valu

[Discuss] Add `location` to generic table spec

Hi folks, I would like to propose to add an optional `location` field to CreateGenricTable Request and LoadGenericTable response. The `location` is the location for the table, which is common to most table formats including Iceberg, Delta, Hudi, csv, parquet etc. The location information is criti

Re: [Discuss] Add `location` to generic table spec

Catalogs, and the URI format seems the standard format used by various Catalog services. Best Regards, Yun On Wed, May 7, 2025 at 4:55 PM yun zou wrote: > Hi Dimitri and Eric, > > Thanks a lot for the feedback! > > For the questions: > - Is one value or many? > It will be one

Re: [Discuss] Add `location` to generic table spec

n explicit field could make things more clear when sharing across engines. Please let me know if there is any point I am missing! Looking forward to your reply and feeback! Best Regards, Yun On Wed, May 7, 2025 at 6:51 PM yun zou wrote: > Hi Dmitri, > > If it's not "all&q

Re: [Discuss] Add `location` to generic table spec

ints about Generic Table > locations in the > Open API spec, I'd be fine with that. > > Cheers, > Dmitri. > > On Mon, May 19, 2025 at 6:46 PM yun zou > wrote: > > > Hi Dmitri, > > > > Thanks for the detailed explanation, I definitely agree we need to

Re: Proposal for Implementation of the Proposed Iceberg REST Spec - Events API

Hi Adnan, That is a really interesting and important work! Implementing event storage through persistent backers totally makes sense to me. For the event buffer, I think it is great to batch the events and reduce access to persistence. However, I am wondering if durability during service start is

Re: [Discuss] Add `location` to generic table spec

n and File based tables they > all > > > have to live in the root location. I'm not sure of any other "file" > based > > > tables where this would be an issue but I'd love to know if someone > else > > > has ideas. I think with the rise in creden

Re: [Discuss] Add `location` to generic table spec

* The location is a base URI (essentially prefix) for all files in a > > generic table. > > * Clients (engines) are responsible for writing files only under the > > specified location. > > * A table, whose files exist outside the declared location, is not > > compliant

Re: [Discuss] Add Policy Privileges and PolicyGrant to Management Spec

+1 (non-binding) Thanks Jonas! On Wed, May 14, 2025 at 12:13 PM Yufei Gu wrote: > +1. Thanks Jonas! > Yufei > > > On Tue, May 13, 2025 at 9:56 PM Honah J. wrote: > > > Hi folks, > > > > I would like to propose extending the management API specification to add > > policy-related privileges and

Re: [Discuss] Add `location` to generic table spec

also get a site page to describe the support for Generic Table and key fields. Best Regards, Yun On Mon, May 19, 2025 at 11:16 PM yun zou wrote: > Hi Dmitri, > > " I do not think those doc comments provide enough visibility to ensure > that the key information > is received b

Re: [Discuss] Add `location` to generic table spec

> This is a stricter requirement than we have for Iceberg tables. Are we really going to enforce this? How will we do it efficiently? If not, let's not put it in the spec. The efficiency is a good point, if we are supporting arbitrary nested namespaces, the efficiency is definitely a concern. Mayb

Re: [Discuss] Add `location` to generic table spec

ables and doesn’t sound > > > correct. > > > > > > > > Similarly, the restriction that you can’t change location is stricter > > > than > > > > what we have for Iceberg. > > > > > > > > Finally, I’m still not sure what the prob

[Discussion] Add RealmContext parameter to loadTasks interface in PolarisMetaStoreManager

Hi Team, In order to get rid of the usage of CallContext.getCurrentContext() call in the getConfiguration implementation of DefaultConfigurationStore, we did one attempt leverage CDI to inject request scoped RealmContext into the application scoped DefaultConfigurationStore. Commit: https://github

Re: [DISCUSS[ Spark Client jars: maven coordinates and shading

, Jun 20, 2025 at 9:47 AM yun zou wrote: > > *-- What is the maven artifact that Spark can automatically pull > (via--packages)* > > Our spark client pulls the following: > > org.apache.polaris#polaris-spark-3.5_2.12 > > org.apache.polaris#polaris-core > > org.apac

Re: [DISCUSS[ Spark Client jars: maven coordinates and shading

*-- What is the maven artifact that Spark can automatically pull (via--packages)* Our spark client pulls the following: org.apache.polaris#polaris-spark-3.5_2.12 org.apache.polaris#polaris-core org.apache.polaris#polaris-api-management-model org.apache.iceberg#iceberg-spark-runtime-3.5_2.12

Re: [DISCUSS[ Spark Client jars: maven coordinates and shading

Hi Dmitri, Thanks for bringing that up! Yes, I think it makes sense to put this in 1.0. I can work on the cherry-pick once everything is addressed. Best Regards, Yun On Fri, Jun 20, 2025 at 7:17 AM Dmitri Bourlatchkov wrote: > Hi All, > > Re: PR [1908] let's use this thread to clarify the pro

Re: [DISCUSS[ Spark Client jars: maven coordinates and shading

.1.0-incubating-SNAPSHOT-sources.jar > > ... but Spark still works with --packages > org.apache.polaris:polaris-spark-3.5_2.12:1.1.0-incubating-SNAPSHOT > > Cheers, > Dmitri. > > On Fri, Jun 20, 2025 at 6:42 PM yun zou > wrote: > > > Hi Dmitri, > > > > I t

Re: [DISCUSS[ Spark Client jars: maven coordinates and shading

ntenance POV. Yet, I do not insist on reworking this. > > Cheers, > Dmitri. > > > On Fri, Jun 20, 2025 at 5:09 PM yun zou > wrote: > > > Hi Dmitri, > > > > Regarding to this question: > > > > > > > > > > *Current doc

Re: [DISCUSS[ Spark Client jars: maven coordinates and shading

Jun 20, 2025 at 1:05 PM Dmitri Bourlatchkov < > di...@apache.org > > > > > >> > wrote: > > >> > > > >> > > Thanks for the quick response, Yun! > > >> > > > > >> > > > org.apache.polaris#polaris-cor

Re: [VOTE] Release Apache Polaris 1.0.0-incubating (RC0)

Hi Dimitri, Regarding polaris-spark-3.5_2.12-1.0.0-incubating.pom, for dependencies like junut-jupyter, mockito-core, those are used by testing and marked as testImplementation in the build file. In the .pom file the optional is marked as true for those dependencies, therefore when using it with -

Re: [VOTE] Release Apache Polaris 1.0.0-incubating (RC0)

by Spark, so this is > not a problem for 1.0. > > Still, why would "testImplementation" dependencies be added as "runtime" > (not as "test") in POM? Should we still fix it on "main"? WDYT? > > Thanks, > Dmitri. > > On Tue, Jun 24, 20

Re: Adding Support for Apache Hudi within Polaris

Hi Rahil, Thanks a lot for working on this! This is definitely one valuable feature improvement for the Spark Client support, I am really looking forward to that! The overall approach looks good to me! Best Regards, Yun On Fri, Jun 13, 2025 at 3:47 PM Yufei Gu wrote: > Thanks for the proposal

Re: [DISCUSS] Injecting RealmContext

d as PolarisCallContext parameters. The > same holds for AtomicOperationMetaStoreManager. > > As such, PolarisMetaStoreManager can be request-scoped beans. I do not see > any reason to keep those objects after the request is done. > > WDYT? > > Thanks, > Dmitri. >

Re: [DISCUSS] Injecting RealmContext

Hi Dmitri, Thanks for bringing that up! I am not an expert with CDI, based my recent experience with CDI, while CDI does help making development simpler under many cases, it does also make things more complicated under some scenarios. Especially when injecting a request scoped object into applica

Re: [DISCUSS] Prepare for 1.0 Release

+1 on the 1.0 release. I have verified manually with the spark client package use case with the current maven repo, I was able to create/insert/drop iceberg and delta table as expected. Best Regards, Yun On Mon, Jun 23, 2025 at 6:32 PM Yufei Gu wrote: > Thanks a lot for everyone working on thi

Re: [VOTE] Release Apache Polaris 1.0.0-incubating (RC0)

+1, verified spark client package use case. On Mon, Jun 23, 2025 at 6:24 PM Yufei Gu wrote: > Hi everyone, > > I propose that we release the following RC as the official > Apache Polaris 1.0.0-incubating release. > > This corresponds to the tag: apache-polaris-1.0.0-incubating-rc0 > * > > https:

Re: [Discuss] Add `location` to generic table spec

fields clearly. If there is no objection for the current plan, we would like to move on for the PR review: https://github.com/apache/polaris/pull/1543/files Best Regards, Yun On Thu, May 22, 2025 at 7:32 PM yun zou wrote: > > This is a stricter requirement than we have for Iceberg table

Re: [Discuss] Add `location` to generic table spec

nd of changes, it looks good to me too! Thanks for > > > working on this. > > > > > > On Tue, Jun 10, 2025 at 4:56 PM Yufei Gu wrote: > > > > > > > Sounds good to me. > > > > Yufei > > > > > > > > > > > > On Tue, Jun 10, 2

Re: [Discuss] Add `location` to generic table spec

Hi Laurent, I do agree that the generic table spec needs to be evolved to provide richer standardization. However, the current support of generic table does provides good amount of values: 1) Polaris can be a centralized catalog service for discovering all tables, not just Iceberg tables. (Of cour

Re: [Discuss] Add `location` to generic table spec

> I mean a doc page similar to [1] that explains what Generic Tables are, how to use them in Spark, how to use them is some other query engine, and most importantly the planned evolution for the Generic Tables API and specification. Yes, we can definitely add a webpage to describe the current guar

Re: [Discuss] Add `location` to generic table spec

> On Wed, Jun 11, 2025 at 5:45 PM yun zou > wrote: > > > > I mean a doc page similar to [1] that explains what Generic Tables are, > > how > > to use them in Spark, how to use them is some other query engine, and > most > > importantly the planned evoluti

Re: [Discussion] Decoupling Helm chart releases (+ nightly build proposal)

On Fri, Jul 18, 2025 at 1:39 PM yun zou wrote: > The purpose of a nightly Helm Chart release is to *quickly* unblock users, > as in this issue: https://github.com/apache/polaris/issues/2030. > > The release process for the nightly Helm Chart would follow the same > approach >

Re: [Discussion] Decoupling Helm chart releases (+ nightly build proposal)

-porting just helm chart fixes there and using the same manual process > as for 1.0.0. This will not require adding a separate source bundle for the > charts (it's part of the normal release already). > > Cheers, > Dmitri. > > On Fri, Jul 18, 2025 at 1:55 PM yun zo

Re: [VOTE] Release Apache Polaris 1.0.0-incubating (RC6)

+ 1 (non-binding) Verified usage of Polaris Spark client plugin with --package and --jars usage. The operations verified: - create/drop namespaces - create/insert/list/load/drop tables Best Regards, Yun On Wed, Jul 2, 2025 at 12:46 PM Dennis Huo wrote: > +1 (binding) > > -Verified all checksum

Re: [ANNOUNCE] Apache Polaris 1.0.0-incubating

This is fantastic news! Huge thanks to you , Yufei! Great work! Best Regards, Yun On Thu, Jul 10, 2025 at 9:11 AM Prashant Singh wrote: > Awesome news ! > Thank you Yufei for driving the release :) ! > > Thanks a ton to the Polaris community for your participation and > contribution; it wouldn

Re: [VOTE] Release Apache Polaris 1.0.0-incubating (RC6)

Agree with Russel. PR 1991 is intended to affect only the bundle JAR usage when using the --jars option. The --packages use case will continue to work as it does today. Since the JAR name already changes with each version update, I don’t believe the JAR name change alone should block the 1.0 relea

Re: [VOTE] Release Apache Polaris 1.0.0-incubating (RC5)

+1 (non-binding) I tested the spark client using both --packages and --jars options, tested operations: 1) create/insert values for both delta and iceberg tables 2) create/drop namespaces 3) create/drop iceberg and delta tables Things turned out to work accordingly. Best Regards, Yun On Mon, Ju

Re: [Discussion] Decoupling Helm chart releases (+ nightly build proposal)

If we decide to adopt an independent release cadence for the Helm chart, it might be more intuitive to host it in a separate repository. While this would increase the effort required to maintain compatibility between Helm chart releases and Polaris releases—particularly around testing and documenta

Re: [VOTE] Release Apache Polaris 1.0.0-incubating (RC2)

Hi Russel, Thanks a lot! I can follow up on this once I am back. Best Regards, Yun On Thu, Jun 26, 2025 at 12:37 PM Russell Spitzer wrote: > PR Fixing up the license issue in the Spark Client Jar - > https://github.com/apache/polaris/pull/1950 > I wasn't able to get the Shadow Jar to do it wit

Re: [Discussion] Decoupling Helm chart releases (+ nightly build proposal)

n Mon, Jul 14, 2025 at 11:38 AM yun zou wrote: > If we decide to adopt an independent release cadence for the Helm > chart, it might > be more intuitive to host it in a separate repository. While this would > increase the > effort required to maintain compatibility between Hel

Re: [Discussion] Decoupling Helm chart releases (+ nightly build proposal)

ghtly release" for helm charts? How will it > work technically? > > Thanks, > Dmitri. > > On Fri, Jul 18, 2025 at 11:54 AM yun zou > wrote: > > > Hi Team, > > > > I'd like to bring this thread back to the top. Aside from the long-term > > p

Re: [Discussion] Decoupling Helm chart releases (+ nightly build proposal)

> > > > > Hi Yun > > > > > > > > > > I would propose to focus on the following: > > > > > 1. We have a "isolated" release cycle for Helm Chart > > > > > 2. We do a Helm Chart release as soon as we need to fix/

Re: [Discussion] Decoupling Helm chart releases (+ nightly build proposal)

hot so > it's not for production). For production, as we plan independent > release cycle for Helm Chart, we should just do a "regular" release. > > Regards > JB > > On Fri, Jul 18, 2025 at 10:39 PM yun zou > wrote: > > > > The purpose of a

Re: [PROPOSAL] Release Apache Polaris 1.0.1-incubating

+1 on 1.0.1 with just helm fix. Best Regards, Yun On Tue, Jul 22, 2025 at 9:05 AM Prashant Singh wrote: > +1 to 1.0.1 with just helm fix ! > > On Tue, Jul 22, 2025 at 7:07 AM Dmitri Bourlatchkov > wrote: > > > A 1.0.1 (patch) release from the 1.0.x branch with just helm fixes SGTM. > > > > Che