Apache Polaris OSS Roadmap Proposal

[Yufei Gu]

Over the past months, we've collaborated with a wide range of
stakeholders—companies, developers, and users—who are invested in the
evolution of Apache Polaris to come up with a community-driven roadmap. It
consolidates those insights into a shared vision, ensuring that our efforts
address the most impactful and widely supported improvements. We appreciate
the valuable feedback and collaboration that have shaped this direction.

The roadmap Items can be broadly classified into several categories such as

   - Core Polaris functions
   - Catalog Federation and Integrations
   - Data Security, Data Governance and Compliance
   - Observability and Reliability
   - AI/ML

Checking the proposal details here,
https://github.com/apache/polaris/discussions/1028


*Note that the features can move in and out of the milestones based on
prioritization, and available resources.*

Yufei

Re: Donate Nessie Iceberg Catalog migrator

[Jean-Baptiste Onofré]

Hi Dmitri

About "evolution plan", I see the catalog migrator tool evolving as a
set of beans/providers that will be used in both CLI, and some server
features (like federated catalogs or "foreign catalogs").
We should not focus too much on catalog migrator as it is today but
more how it will be tomorrow.

That's why I'm more in favor of preparing the field and donating as a
module in the Polaris repo.

Regards
JB

On Thu, Feb 20, 2025 at 6:38 AM Dmitri Bourlatchkov  wrote:
>
> +1 to accept the catalog migrator tool.
>
> I support inviting Ajantha as a committer.
>
> As to the source location, I tend to think that a separate repo makes sense
> with the current state of the code, but I also agree that the overhead of
> that may be too much, given that the codebase is small. I'm fine with
> either a separate repo or a new module in the current Polaris repo.
>
> What is the general plan for the evolution of the migrator tool? Are we
> talking about integrating it into Polaris Servers or will it remain a
> standalone tool as it is now?
>
> Thanks,
> Dmitri.
>
> On Wed, Feb 19, 2025 at 11:39 AM Jean-Baptiste Onofré 
> wrote:
>
> > Hi folks,
> >
> > Let me try to sum-up this topic.
> >
> > 1. Catalog Migration landing
> > It seems we have a preference to land catalog-migrator as a module on
> > the main polaris repo.
> > Robert expressed comments about CI, release cycle, dependencies.
> >
> > My view on that is that the purpose of the catalog-migrator is to
> > evolve, and could become a key component for features like federated
> > catalogs.
> > Due to that, I think we can consider catalog-migrator as a module,
> > integrated in the Polaris CLI, or in the Polaris server,
> >
> > Robert, does it work for you ?
> >
> > 2. Code/PR prep
> > I propose to work directly with Ajantha (main contributor of the
> > catalog-migrator) to prepare the code heading to a PR. We need:
> > - integrate in Polaris repo and gradle
> > - rename all packages to use org.apache.polaris
> > - add ASF header in all files
> > - refactore cli to use polaris style/naming
> > - refactore intTest to use Polaris instead of Nessie
> > - check the dependencies in the cli uber jar (hadoop, hive, ...) and
> > cleanup LICENSE/NOTICE there
> > - update README and cleanup other files
> > It should be pretty fast and we should be able to create a PR for
> > review/donation.
> >
> > Thoughts ?
> >
> > Regards
> > JB
> >
> > On Tue, Feb 11, 2025 at 7:45 PM Jean-Baptiste Onofré 
> > wrote:
> > >
> > > Hi folks,
> > >
> > > As discussed with some of you, we would like to propose donating the
> > > Nessie Iceberg Catalog migrator tool
> > > (https://github.com/projectnessie/iceberg-catalog-migrator) to Apache
> > > Polaris.
> > >
> > > A SGA has been already signed in case we accept the donation.
> > >
> > > In terms of donation, I propose the following:
> > > - the iceberg-catalog-migration can land in a separate Polaris repo
> > > (as it is today for Nessie):
> > > https://github.com/apache/polaris-catalog-migration or land as a
> > > module in polaris repo directly
> > > - we need to prepare the donation by changing the package names, etc
> > > - I would suggest to consider inviting one of the main contributor of
> > > Catalog Migrator (ajantha-bhat) as Polaris committer
> > >
> > > WDYT ? Do we accept the iceberg-catalog-migrator tool in Polaris (we
> > > can do a formal vote if we don't have obvious consensus) ?
> > >
> > > Regards
> > > JB
> >

Re: [VOTE] Release Apache Polaris 0.9.0-incubating (rc6)

[Robert Stupp]

+1 (binding)

On 17.02.25 21:27, Jean-Baptiste Onofré wrote:

Hi folks,

After the vote on the incubator general mailing list, we fixed the
DISCLAIMER content and cleaned up the NOTICE file.

This is the vote for Apache Polaris 0.9.0-incubating rc6.

* This corresponds to the tag: apache-polaris-0.9.0-incubating-rc6
* https://github.com/apache/polaris/tree/apache-polaris-0.9.0-incubating-rc6
* 
https://github.com/apache/polaris/tree/4b18ec065ff16f74b11bc85fdc6ea9036eca7274

The release source tarball, signature, and checksums are here:
* https://dist.apache.org/repos/dist/dev/incubator/polaris/0.9.0-incubating/

You can find the KEYS file here:
* https://dist.apache.org/repos/dist/release/incubator/polaris/KEYS

NB: this release doesn't provide any binary distribution, it provides
source distribution only.

Please download, verify, and test.

Please vote in the next 72 hours.
[ ] +1 Release this as Apache polaris 0.9.0-incubating
[ ] +0
[ ] -1 Do not release this because...

Only PPMC members and mentors have binding votes, but other community
members are encouraged to cast non-binding votes. This vote will pass
if there are
3 binding +1 votes and more binding +1 votes than -1 votes.

NB: if this vote passes, a new vote will be started on the Incubator
general mailing list.

Thanks
Regards
JB


--
Robert Stupp
@snazy

Re: Preparing 0.9.0 rc6

[Robert Stupp]

Guys, this isn't the VOTE  thread ;)

On 19.02.25 18:40, Jean-Baptiste Onofré wrote:

+1 (binding)

Regards
JB

On Mon, Feb 17, 2025 at 8:36 PM Jean-Baptiste Onofré  wrote:

Hi folks

We merged the fix about DISCLAIMER and NOTICE.

I will proceed with 0.9.0 rc6 vote.

Stay tuned !

Thanks
Regards
JB


--
Robert Stupp
@snazy

Re: Local Laptop Starter Kit::Open to Contribution

[Jean-Baptiste Onofré]

Good point about lab.

I would prefer polaris-examples than because it can host more than
just quickstarts.

Regards
JB

On Thu, Feb 20, 2025 at 8:37 AM Kamesh Sampath
 wrote:
>
> -1 on labs as it seem to be more of R&D kind of thing, since it more aligns 
> with quickstart/get-started kind of things wondering polaris-examples or 
> polaris-quickstarts could be apt as well. If polaris-quickstarts is doable it 
> can also act as house of all future demos/quickstarts.
> 
> From: Russell Spitzer 
> Sent: Wednesday, February 19, 2025 12:43 PM
> To: dev@polaris.apache.org 
> Subject: Re: Local Laptop Starter Kit::Open to Contribution
>
> I think this would be a great thing to include as well! Polaris examples
> seems like a fine place to put it, not sure labs is as appropriate since
> wouldn’t be using the repo for experiments
>
> On Wed, Feb 19, 2025 at 12:54 AM Jean-Baptiste Onofré 
> wrote:
>
> > Hi Kamesh
> >
> > Thanks for sharing ! That's cool !
> >
> > Maybe we could have a repo dedicated to this kind of resource,
> > something like 
> > https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fpolaris-examples&data=05%7C02%7C%7Ce2680bff76014862b30908dd50b5232c%7C84df9e7fe9f640afb435%7C1%7C0%7C638755461085866902%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=uAO3KjcOZ5p3A0jIw3nyLMYjyivfH1xETVcJFzl4RWA%3D&reserved=0
> >  or
> > polaris-lab ?
> >
> > Regards
> > JB
> >
> > On Tue, Feb 18, 2025 at 6:23 PM Kamesh Sampath
> >  wrote:
> > >
> > > Hi,
> > >
> > > I build this starter kit to help kickstarted with Apache Polaris on your
> > local laptop with AWS s3 emulator like localstack with automation scripts
> > that can be improved and used beyond local development environment.
> > >
> > > Git Repo: 
> > > https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FSnowflake-Labs%2Fpolaris-local-forge&data=05%7C02%7C%7Ce2680bff76014862b30908dd50b5232c%7C84df9e7fe9f640afb435%7C1%7C0%7C638755461085882807%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=dzNIRTXDiW7F1C3Mg66DcUEzloHq%2BQOapaV%2BHgzanO8%3D&reserved=0
> > >
> > > Quick start on how that works - 
> > > https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fyoutu.be%2FLvIUv3JtUNs&data=05%7C02%7C%7Ce2680bff76014862b30908dd50b5232c%7C84df9e7fe9f640afb435%7C1%7C0%7C638755461085890895%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=uNeYfShX1WQTMNzgnKAST1jcoe1lMcwpoqh836a0v1U%3D&reserved=0
> > >
> > >
> > > I would like to contribute to Apache Polaris repository, but don't want
> > to mess with root apache/polaris.
> > >
> > > Love to hear the feedback and thoughts and potential path to
> > contribution if it makes value,
> > >
> > > Best,
> > > Kamesh
> >

Re: Local Laptop Starter Kit::Open to Contribution

[Kamesh Sampath]

Sounds good to me.

Get Outlook for iOS

From: Jean-Baptiste Onofré 
Sent: Thursday, February 20, 2025 2:28:27 PM
To: dev@polaris.apache.org 
Subject: Re: Local Laptop Starter Kit::Open to Contribution

Good point about lab.

I would prefer polaris-examples than because it can host more than
just quickstarts.

Regards
JB

On Thu, Feb 20, 2025 at 8:37 AM Kamesh Sampath
 wrote:
>
> -1 on labs as it seem to be more of R&D kind of thing, since it more aligns 
> with quickstart/get-started kind of things wondering polaris-examples or 
> polaris-quickstarts could be apt as well. If polaris-quickstarts is doable it 
> can also act as house of all future demos/quickstarts.
> 
> From: Russell Spitzer 
> Sent: Wednesday, February 19, 2025 12:43 PM
> To: dev@polaris.apache.org 
> Subject: Re: Local Laptop Starter Kit::Open to Contribution
>
> I think this would be a great thing to include as well! Polaris examples
> seems like a fine place to put it, not sure labs is as appropriate since
> wouldn’t be using the repo for experiments
>
> On Wed, Feb 19, 2025 at 12:54 AM Jean-Baptiste Onofré 
> wrote:
>
> > Hi Kamesh
> >
> > Thanks for sharing ! That's cool !
> >
> > Maybe we could have a repo dedicated to this kind of resource,
> > something like 
> > https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fpolaris-examples&data=05%7C02%7C%7C8d90ee8275284a94cc5208dd518d0290%7C84df9e7fe9f640afb435%7C1%7C0%7C638756388260153540%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=7veY1OdZGLgD7fIGl9DauvuTvaMZuVgdBTCsO9nXMgE%3D&reserved=0
> >  or
> > polaris-lab ?
> >
> > Regards
> > JB
> >
> > On Tue, Feb 18, 2025 at 6:23 PM Kamesh Sampath
> >  wrote:
> > >
> > > Hi,
> > >
> > > I build this starter kit to help kickstarted with Apache Polaris on your
> > local laptop with AWS s3 emulator like localstack with automation scripts
> > that can be improved and used beyond local development environment.
> > >
> > > Git Repo: 
> > > https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FSnowflake-Labs%2Fpolaris-local-forge&data=05%7C02%7C%7C8d90ee8275284a94cc5208dd518d0290%7C84df9e7fe9f640afb435%7C1%7C0%7C638756388260192658%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=v0FzoB%2B6S%2FYMsRfGqE04u5gr1JfghnLCCbRRTgyqKhw%3D&reserved=0
> > >
> > > Quick start on how that works - 
> > > https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fyoutu.be%2FLvIUv3JtUNs&data=05%7C02%7C%7C8d90ee8275284a94cc5208dd518d0290%7C84df9e7fe9f640afb435%7C1%7C0%7C638756388260220607%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=lECcu68N7XOyZ0blWCZJWKTjGNJXdvhhDLNx1e9Pw9w%3D&reserved=0
> > >
> > >
> > > I would like to contribute to Apache Polaris repository, but don't want
> > to mess with root apache/polaris.
> > >
> > > Love to hear the feedback and thoughts and potential path to
> > contribution if it makes value,
> > >
> > > Best,
> > > Kamesh
> >

[VOTE] Release Apache Polaris (incubating) 0.9.0-rc6

[Jean-Baptiste Onofré]

Hi folks,

This is a call for the vote to release Apache Polaris (incubating) 0.9.0-rc6.

The Polaris community has voted and approved Apache Polaris
(incubating) 0.9.0-rc6.
We are now asking the IPMC to review and vote for this release.

Polaris community vote thread:
https://lists.apache.org/thread/7jxoswynzpnhsmbt77t4klc7rp983o66

Polaris community vote result:
https://lists.apache.org/thread/3ymqrtlvtcq1pgtd14ghn2n158y6yl5t

* This corresponds to the tag: apache-polaris-0.9.0-incubating-rc6
* https://github.com/apache/polaris/tree/apache-polaris-0.9.0-incubating-rc6
*  
https://github.com/apache/polaris/tree/4b18ec065ff16f74b11bc85fdc6ea9036eca7274

The release source tarball, signature, and checksums are here:
* https://dist.apache.org/repos/dist/dev/incubator/polaris/0.9.0-incubating/

You can find the KEYS file here:
* https://dist.apache.org/repos/dist/release/incubator/polaris/KEYS

The vote will be open for at least 72 hours or until the necessary
number of votes are reached.

Please vote accordingly:

[ ] +1 Release this as Apache Polaris (incubating) 0.9.0
[ ] +0
[ ] -1 Do not release this because...

This vote will pass if there are 3 binding +1 votes and more binding
+1 votes than -1 votes.

Thanks,
Regards
JB

Re: [VOTE] Release Apache Polaris 0.9.0-incubating (rc6)

[Jean-Baptiste Onofré]

Hi Ryan

I agree, and I thought it's what I did, but it's not: my bad.

I propose to move forward with the release to the IPMC and we will
cleanup NOTICE for the next release.

Thanks !
Regards
JB

On Thu, Feb 20, 2025 at 7:22 PM rdb...@gmail.com  wrote:
>
> +1
>
> The update to DISCLAIMER and NOTICE look okay to me, although I don't think 
> that the Nessie NOTICE content is necessary. That information, which is just 
> the copyright, is already in LICENSE to document the parts that are licensed.
>
> On Thu, Feb 20, 2025 at 6:58 AM Russell Spitzer  
> wrote:
>>
>> +1
>>
>> On Thu, Feb 20, 2025 at 6:06 AM Jean-Baptiste Onofré 
>> wrote:
>>
>> > +1 (binding)
>> >
>> > Regards
>> > JB
>> >
>> > Le lun. 17 févr. 2025 à 21:27, Jean-Baptiste Onofré  a
>> > écrit :
>> >
>> > > Hi folks,
>> > >
>> > > After the vote on the incubator general mailing list, we fixed the
>> > > DISCLAIMER content and cleaned up the NOTICE file.
>> > >
>> > > This is the vote for Apache Polaris 0.9.0-incubating rc6.
>> > >
>> > > * This corresponds to the tag: apache-polaris-0.9.0-incubating-rc6
>> > > *
>> > >
>> > https://github.com/apache/polaris/tree/apache-polaris-0.9.0-incubating-rc6
>> > > *
>> > >
>> > https://github.com/apache/polaris/tree/4b18ec065ff16f74b11bc85fdc6ea9036eca7274
>> > >
>> > > The release source tarball, signature, and checksums are here:
>> > > *
>> > >
>> > https://dist.apache.org/repos/dist/dev/incubator/polaris/0.9.0-incubating/
>> > >
>> > > You can find the KEYS file here:
>> > > * https://dist.apache.org/repos/dist/release/incubator/polaris/KEYS
>> > >
>> > > NB: this release doesn't provide any binary distribution, it provides
>> > > source distribution only.
>> > >
>> > > Please download, verify, and test.
>> > >
>> > > Please vote in the next 72 hours.
>> > > [ ] +1 Release this as Apache polaris 0.9.0-incubating
>> > > [ ] +0
>> > > [ ] -1 Do not release this because...
>> > >
>> > > Only PPMC members and mentors have binding votes, but other community
>> > > members are encouraged to cast non-binding votes. This vote will pass
>> > > if there are
>> > > 3 binding +1 votes and more binding +1 votes than -1 votes.
>> > >
>> > > NB: if this vote passes, a new vote will be started on the Incubator
>> > > general mailing list.
>> > >
>> > > Thanks
>> > > Regards
>> > > JB
>> > >
>> >

[RESULT][VOTE] Release Apache Polaris 0.9.0-incubating (rc6)

[Jean-Baptiste Onofré]

Hi folks,

This vote passed with the following result:

+1 (binding): Robert Stupp, JB Onofré, Russell Spitzer, Ryan Blue
+1 (non binding): Dmitri Bourlatchkov

I'm now moving the vote to gene...@incubator.apache.org mailing list.

Thanks everyone for your vote!

Regards
JB

On Mon, Feb 17, 2025 at 9:27 PM Jean-Baptiste Onofré  wrote:
>
> Hi folks,
>
> After the vote on the incubator general mailing list, we fixed the
> DISCLAIMER content and cleaned up the NOTICE file.
>
> This is the vote for Apache Polaris 0.9.0-incubating rc6.
>
> * This corresponds to the tag: apache-polaris-0.9.0-incubating-rc6
> * https://github.com/apache/polaris/tree/apache-polaris-0.9.0-incubating-rc6
> * 
> https://github.com/apache/polaris/tree/4b18ec065ff16f74b11bc85fdc6ea9036eca7274
>
> The release source tarball, signature, and checksums are here:
> * https://dist.apache.org/repos/dist/dev/incubator/polaris/0.9.0-incubating/
>
> You can find the KEYS file here:
> * https://dist.apache.org/repos/dist/release/incubator/polaris/KEYS
>
> NB: this release doesn't provide any binary distribution, it provides
> source distribution only.
>
> Please download, verify, and test.
>
> Please vote in the next 72 hours.
> [ ] +1 Release this as Apache polaris 0.9.0-incubating
> [ ] +0
> [ ] -1 Do not release this because...
>
> Only PPMC members and mentors have binding votes, but other community
> members are encouraged to cast non-binding votes. This vote will pass
> if there are
> 3 binding +1 votes and more binding +1 votes than -1 votes.
>
> NB: if this vote passes, a new vote will be started on the Incubator
> general mailing list.
>
> Thanks
> Regards
> JB

Re: Donate Nessie Iceberg Catalog migrator

[Jean-Baptiste Onofré]

OK, let's move forward then

I will prepare https://github.com/apache/polaris-catalogs-migrator
repository as a transition phase.

> I also support inviting Ajantha as a committer.
That will be a separate discussion with PPMC.

> Still really prefer a separate repository, at least for now. We can always 
> merge later. It also makes the "migration to ASF" easier.
>
> The "Nessie Iceberg-catalog-migrator tool" has been built to support 
> migrations from any catalog to any other catalog. This means, that the test 
> matrix is quite complex and I expect it to become even more complex and time 
> consuming. IMHO there's no need to "bother" Polaris "main CI" for every PR 
> with catalog-migrator tests. I would also expect a different release cycle - 
> no need to tie both together.

Understood, let's use a specific repo for now.

> Better use org.apache.polaris.catalogmigrator as the base group ID.

Yes, that was the intent indeed.

> This requires https://github.com/apache/polaris/pull/785. I'd also prefer to 
> keep the ITs against Nessie and add Polaris.

The itests will probably evolve a lot depending of the new use of
catalogs-migrator.
The itests bring a bunch of dependencies (Nessie, Hive, ...). As we
are talking about test dependencies, that's OK.

>
> We don't know yet how these (and other features) will look like and not how 
> it'll be related to the donated tool.
>

I'm very enthusiastic about catalogs-migrator: I see a lot of potential :)
For instance, the first "obvious" move is probably catalogs-migrator
will evolve from a "standalone tool" to a library that we can use in
Polaris server (federated catalog, etc).

>
> Please coordinate the migration from the source repo with me. We need to 
> clean some things up on the projectnessie Github org side first.
>

Ack, I will ping you :)

For the rest of the community, no objection to starting with
https://github.com/apache/polaris-catalogs-migrator ?

Regards
JB

>
> Robert
>
> On 20.02.25 09:57, Jean-Baptiste Onofré wrote:
> > Hi Dmitri
> >
> > About "evolution plan", I see the catalog migrator tool evolving as a
> > set of beans/providers that will be used in both CLI, and some server
> > features (like federated catalogs or "foreign catalogs").
> > We should not focus too much on catalog migrator as it is today but
> > more how it will be tomorrow.
> >
> > That's why I'm more in favor of preparing the field and donating as a
> > module in the Polaris repo.
> >
> > Regards
> > JB
> >
> > On Thu, Feb 20, 2025 at 6:38 AM Dmitri Bourlatchkov  
> > wrote:
> >> +1 to accept the catalog migrator tool.
> >>
> >> I support inviting Ajantha as a committer.
> >>
> >> As to the source location, I tend to think that a separate repo makes sense
> >> with the current state of the code, but I also agree that the overhead of
> >> that may be too much, given that the codebase is small. I'm fine with
> >> either a separate repo or a new module in the current Polaris repo.
> >>
> >> What is the general plan for the evolution of the migrator tool? Are we
> >> talking about integrating it into Polaris Servers or will it remain a
> >> standalone tool as it is now?
> >>
> >> Thanks,
> >> Dmitri.
> >>
> >> On Wed, Feb 19, 2025 at 11:39 AM Jean-Baptiste Onofré 
> >> wrote:
> >>
> >>> Hi folks,
> >>>
> >>> Let me try to sum-up this topic.
> >>>
> >>> 1. Catalog Migration landing
> >>> It seems we have a preference to land catalog-migrator as a module on
> >>> the main polaris repo.
> >>> Robert expressed comments about CI, release cycle, dependencies.
> >>>
> >>> My view on that is that the purpose of the catalog-migrator is to
> >>> evolve, and could become a key component for features like federated
> >>> catalogs.
> >>> Due to that, I think we can consider catalog-migrator as a module,
> >>> integrated in the Polaris CLI, or in the Polaris server,
> >>>
> >>> Robert, does it work for you ?
> >>>
> >>> 2. Code/PR prep
> >>> I propose to work directly with Ajantha (main contributor of the
> >>> catalog-migrator) to prepare the code heading to a PR. We need:
> >>> - integrate in Polaris repo and gradle
> >>> - rename all packages to use org.apache.polaris
> >>> - add ASF header in all files
> >>> - refactore cli to use polaris style/naming
> >>> - refactore intTest to use Polaris instead of Nessie
> >>> - check the dependencies in the cli uber jar (hadoop, hive, ...) and
> >>> cleanup LICENSE/NOTICE there
> >>> - update README and cleanup other files
> >>> It should be pretty fast and we should be able to create a PR for
> >>> review/donation.
> >>>
> >>> Thoughts ?
> >>>
> >>> Regards
> >>> JB
> >>>
> >>> On Tue, Feb 11, 2025 at 7:45 PM Jean-Baptiste Onofré 
> >>> wrote:
>  Hi folks,
> 
>  As discussed with some of you, we would like to propose donating the
>  Nessie Iceberg Catalog migrator tool
>  (https://github.com/projectnessie/iceberg-catalog-migrator) to Apache
>  Polaris.
> 
>  A SGA has been already signed in case we accept the dona

Re: Preparing 0.9.0 rc6

[Jean-Baptiste Onofré]

My bad :) I mixed up on my own emails ;)

Thanks for pointing out ;)

Le jeu. 20 févr. 2025 à 12:38, Robert Stupp  a écrit :

> Guys, this isn't the VOTE  thread ;)
>
> On 19.02.25 18:40, Jean-Baptiste Onofré wrote:
> > +1 (binding)
> >
> > Regards
> > JB
> >
> > On Mon, Feb 17, 2025 at 8:36 PM Jean-Baptiste Onofré 
> wrote:
> >> Hi folks
> >>
> >> We merged the fix about DISCLAIMER and NOTICE.
> >>
> >> I will proceed with 0.9.0 rc6 vote.
> >>
> >> Stay tuned !
> >>
> >> Thanks
> >> Regards
> >> JB
>
> --
> Robert Stupp
> @snazy
>
>

Re: [VOTE] Release Apache Polaris 0.9.0-incubating (rc6)

[Russell Spitzer]

+1

On Thu, Feb 20, 2025 at 6:06 AM Jean-Baptiste Onofré 
wrote:

> +1 (binding)
>
> Regards
> JB
>
> Le lun. 17 févr. 2025 à 21:27, Jean-Baptiste Onofré  a
> écrit :
>
> > Hi folks,
> >
> > After the vote on the incubator general mailing list, we fixed the
> > DISCLAIMER content and cleaned up the NOTICE file.
> >
> > This is the vote for Apache Polaris 0.9.0-incubating rc6.
> >
> > * This corresponds to the tag: apache-polaris-0.9.0-incubating-rc6
> > *
> >
> https://github.com/apache/polaris/tree/apache-polaris-0.9.0-incubating-rc6
> > *
> >
> https://github.com/apache/polaris/tree/4b18ec065ff16f74b11bc85fdc6ea9036eca7274
> >
> > The release source tarball, signature, and checksums are here:
> > *
> >
> https://dist.apache.org/repos/dist/dev/incubator/polaris/0.9.0-incubating/
> >
> > You can find the KEYS file here:
> > * https://dist.apache.org/repos/dist/release/incubator/polaris/KEYS
> >
> > NB: this release doesn't provide any binary distribution, it provides
> > source distribution only.
> >
> > Please download, verify, and test.
> >
> > Please vote in the next 72 hours.
> > [ ] +1 Release this as Apache polaris 0.9.0-incubating
> > [ ] +0
> > [ ] -1 Do not release this because...
> >
> > Only PPMC members and mentors have binding votes, but other community
> > members are encouraged to cast non-binding votes. This vote will pass
> > if there are
> > 3 binding +1 votes and more binding +1 votes than -1 votes.
> >
> > NB: if this vote passes, a new vote will be started on the Incubator
> > general mailing list.
> >
> > Thanks
> > Regards
> > JB
> >
>

Re: Donate Nessie Iceberg Catalog migrator

[Ajantha Bhat]

Thanks, everyone, for your interest in the catalog migrator tool!
I'm excited to collaborate with the Apache Polaris community on
contributing to it.

- Ajantha

On Thu, Feb 20, 2025 at 6:04 PM Jean-Baptiste Onofré 
wrote:

> OK, let's move forward then
>
> I will prepare https://github.com/apache/polaris-catalogs-migrator
> repository as a transition phase.
>
> > I also support inviting Ajantha as a committer.
> That will be a separate discussion with PPMC.
>
> > Still really prefer a separate repository, at least for now. We can
> always merge later. It also makes the "migration to ASF" easier.
> >
> > The "Nessie Iceberg-catalog-migrator tool" has been built to support
> migrations from any catalog to any other catalog. This means, that the test
> matrix is quite complex and I expect it to become even more complex and
> time consuming. IMHO there's no need to "bother" Polaris "main CI" for
> every PR with catalog-migrator tests. I would also expect a different
> release cycle - no need to tie both together.
>
> Understood, let's use a specific repo for now.
>
> > Better use org.apache.polaris.catalogmigrator as the base group ID.
>
> Yes, that was the intent indeed.
>
> > This requires https://github.com/apache/polaris/pull/785. I'd also
> prefer to keep the ITs against Nessie and add Polaris.
>
> The itests will probably evolve a lot depending of the new use of
> catalogs-migrator.
> The itests bring a bunch of dependencies (Nessie, Hive, ...). As we
> are talking about test dependencies, that's OK.
>
> >
> > We don't know yet how these (and other features) will look like and not
> how it'll be related to the donated tool.
> >
>
> I'm very enthusiastic about catalogs-migrator: I see a lot of potential :)
> For instance, the first "obvious" move is probably catalogs-migrator
> will evolve from a "standalone tool" to a library that we can use in
> Polaris server (federated catalog, etc).
>
> >
> > Please coordinate the migration from the source repo with me. We need to
> clean some things up on the projectnessie Github org side first.
> >
>
> Ack, I will ping you :)
>
> For the rest of the community, no objection to starting with
> https://github.com/apache/polaris-catalogs-migrator ?
>
> Regards
> JB
>
> >
> > Robert
> >
> > On 20.02.25 09:57, Jean-Baptiste Onofré wrote:
> > > Hi Dmitri
> > >
> > > About "evolution plan", I see the catalog migrator tool evolving as a
> > > set of beans/providers that will be used in both CLI, and some server
> > > features (like federated catalogs or "foreign catalogs").
> > > We should not focus too much on catalog migrator as it is today but
> > > more how it will be tomorrow.
> > >
> > > That's why I'm more in favor of preparing the field and donating as a
> > > module in the Polaris repo.
> > >
> > > Regards
> > > JB
> > >
> > > On Thu, Feb 20, 2025 at 6:38 AM Dmitri Bourlatchkov 
> wrote:
> > >> +1 to accept the catalog migrator tool.
> > >>
> > >> I support inviting Ajantha as a committer.
> > >>
> > >> As to the source location, I tend to think that a separate repo makes
> sense
> > >> with the current state of the code, but I also agree that the
> overhead of
> > >> that may be too much, given that the codebase is small. I'm fine with
> > >> either a separate repo or a new module in the current Polaris repo.
> > >>
> > >> What is the general plan for the evolution of the migrator tool? Are
> we
> > >> talking about integrating it into Polaris Servers or will it remain a
> > >> standalone tool as it is now?
> > >>
> > >> Thanks,
> > >> Dmitri.
> > >>
> > >> On Wed, Feb 19, 2025 at 11:39 AM Jean-Baptiste Onofré <
> j...@nanthrax.net>
> > >> wrote:
> > >>
> > >>> Hi folks,
> > >>>
> > >>> Let me try to sum-up this topic.
> > >>>
> > >>> 1. Catalog Migration landing
> > >>> It seems we have a preference to land catalog-migrator as a module on
> > >>> the main polaris repo.
> > >>> Robert expressed comments about CI, release cycle, dependencies.
> > >>>
> > >>> My view on that is that the purpose of the catalog-migrator is to
> > >>> evolve, and could become a key component for features like federated
> > >>> catalogs.
> > >>> Due to that, I think we can consider catalog-migrator as a module,
> > >>> integrated in the Polaris CLI, or in the Polaris server,
> > >>>
> > >>> Robert, does it work for you ?
> > >>>
> > >>> 2. Code/PR prep
> > >>> I propose to work directly with Ajantha (main contributor of the
> > >>> catalog-migrator) to prepare the code heading to a PR. We need:
> > >>> - integrate in Polaris repo and gradle
> > >>> - rename all packages to use org.apache.polaris
> > >>> - add ASF header in all files
> > >>> - refactore cli to use polaris style/naming
> > >>> - refactore intTest to use Polaris instead of Nessie
> > >>> - check the dependencies in the cli uber jar (hadoop, hive, ...) and
> > >>> cleanup LICENSE/NOTICE there
> > >>> - update README and cleanup other files
> > >>> It should be pretty fast and we should be able to create a PR for
> > >>> re

Re: [VOTE] Release Apache Polaris 0.9.0-incubating (rc6)

[rdb...@gmail.com]

+1

The update to DISCLAIMER and NOTICE look okay to me, although I don't think
that the Nessie NOTICE content is necessary. That information, which is
just the copyright, is already in LICENSE to document the parts that are
licensed.

On Thu, Feb 20, 2025 at 6:58 AM Russell Spitzer 
wrote:

> +1
>
> On Thu, Feb 20, 2025 at 6:06 AM Jean-Baptiste Onofré 
> wrote:
>
> > +1 (binding)
> >
> > Regards
> > JB
> >
> > Le lun. 17 févr. 2025 à 21:27, Jean-Baptiste Onofré  a
> > écrit :
> >
> > > Hi folks,
> > >
> > > After the vote on the incubator general mailing list, we fixed the
> > > DISCLAIMER content and cleaned up the NOTICE file.
> > >
> > > This is the vote for Apache Polaris 0.9.0-incubating rc6.
> > >
> > > * This corresponds to the tag: apache-polaris-0.9.0-incubating-rc6
> > > *
> > >
> >
> https://github.com/apache/polaris/tree/apache-polaris-0.9.0-incubating-rc6
> > > *
> > >
> >
> https://github.com/apache/polaris/tree/4b18ec065ff16f74b11bc85fdc6ea9036eca7274
> > >
> > > The release source tarball, signature, and checksums are here:
> > > *
> > >
> >
> https://dist.apache.org/repos/dist/dev/incubator/polaris/0.9.0-incubating/
> > >
> > > You can find the KEYS file here:
> > > * https://dist.apache.org/repos/dist/release/incubator/polaris/KEYS
> > >
> > > NB: this release doesn't provide any binary distribution, it provides
> > > source distribution only.
> > >
> > > Please download, verify, and test.
> > >
> > > Please vote in the next 72 hours.
> > > [ ] +1 Release this as Apache polaris 0.9.0-incubating
> > > [ ] +0
> > > [ ] -1 Do not release this because...
> > >
> > > Only PPMC members and mentors have binding votes, but other community
> > > members are encouraged to cast non-binding votes. This vote will pass
> > > if there are
> > > 3 binding +1 votes and more binding +1 votes than -1 votes.
> > >
> > > NB: if this vote passes, a new vote will be started on the Incubator
> > > general mailing list.
> > >
> > > Thanks
> > > Regards
> > > JB
> > >
> >
>

Re: [VOTE] Release Apache Polaris 0.9.0-incubating (rc6)

[Dmitri Bourlatchkov]

+1 (nb)

Verified:
* checksum
* signature
* ./gradlew build

Cheers,
Dmitri.

On Mon, Feb 17, 2025 at 3:28 PM Jean-Baptiste Onofré 
wrote:

> Hi folks,
>
> After the vote on the incubator general mailing list, we fixed the
> DISCLAIMER content and cleaned up the NOTICE file.
>
> This is the vote for Apache Polaris 0.9.0-incubating rc6.
>
> * This corresponds to the tag: apache-polaris-0.9.0-incubating-rc6
> *
> https://github.com/apache/polaris/tree/apache-polaris-0.9.0-incubating-rc6
> *
> https://github.com/apache/polaris/tree/4b18ec065ff16f74b11bc85fdc6ea9036eca7274
>
> The release source tarball, signature, and checksums are here:
> *
> https://dist.apache.org/repos/dist/dev/incubator/polaris/0.9.0-incubating/
>
> You can find the KEYS file here:
> * https://dist.apache.org/repos/dist/release/incubator/polaris/KEYS
>
> NB: this release doesn't provide any binary distribution, it provides
> source distribution only.
>
> Please download, verify, and test.
>
> Please vote in the next 72 hours.
> [ ] +1 Release this as Apache polaris 0.9.0-incubating
> [ ] +0
> [ ] -1 Do not release this because...
>
> Only PPMC members and mentors have binding votes, but other community
> members are encouraged to cast non-binding votes. This vote will pass
> if there are
> 3 binding +1 votes and more binding +1 votes than -1 votes.
>
> NB: if this vote passes, a new vote will be started on the Incubator
> general mailing list.
>
> Thanks
> Regards
> JB
>

Re: [VOTE] Release Apache Polaris 0.9.0-incubating (rc6)

[Jean-Baptiste Onofré]

+1 (binding)

Regards
JB

Le lun. 17 févr. 2025 à 21:27, Jean-Baptiste Onofré  a
écrit :

> Hi folks,
>
> After the vote on the incubator general mailing list, we fixed the
> DISCLAIMER content and cleaned up the NOTICE file.
>
> This is the vote for Apache Polaris 0.9.0-incubating rc6.
>
> * This corresponds to the tag: apache-polaris-0.9.0-incubating-rc6
> *
> https://github.com/apache/polaris/tree/apache-polaris-0.9.0-incubating-rc6
> *
> https://github.com/apache/polaris/tree/4b18ec065ff16f74b11bc85fdc6ea9036eca7274
>
> The release source tarball, signature, and checksums are here:
> *
> https://dist.apache.org/repos/dist/dev/incubator/polaris/0.9.0-incubating/
>
> You can find the KEYS file here:
> * https://dist.apache.org/repos/dist/release/incubator/polaris/KEYS
>
> NB: this release doesn't provide any binary distribution, it provides
> source distribution only.
>
> Please download, verify, and test.
>
> Please vote in the next 72 hours.
> [ ] +1 Release this as Apache polaris 0.9.0-incubating
> [ ] +0
> [ ] -1 Do not release this because...
>
> Only PPMC members and mentors have binding votes, but other community
> members are encouraged to cast non-binding votes. This vote will pass
> if there are
> 3 binding +1 votes and more binding +1 votes than -1 votes.
>
> NB: if this vote passes, a new vote will be started on the Incubator
> general mailing list.
>
> Thanks
> Regards
> JB
>

Re: Donate Nessie Iceberg Catalog migrator

[Robert Stupp]

Hi,

I also support inviting Ajantha as a committer.

Still really prefer a separate repository, at least for now. We can always merge later. 
It also makes the "migration to ASF" easier.

The "Nessie Iceberg-catalog-migrator tool" has been built to support migrations from any catalog to 
any other catalog. This means, that the test matrix is quite complex and I expect it to become even more 
complex and time consuming. IMHO there's no need to "bother" Polaris "main CI" for every 
PR with catalog-migrator tests. I would also expect a different release cycle - no need to tie both together.


rename all packages to use org.apache.polaris


Better use org.apache.polaris.catalogmigrator as the base group ID.


refactore intTest to use Polaris instead of Nessie


This requires https://github.com/apache/polaris/pull/785. I'd also prefer to 
keep the ITs against Nessie and add Polaris.


key component for features like federated catalogs.


We don't know yet how these (and other features) will look like and not how 
it'll be related to the donated tool.


Please coordinate the migration from the source repo with me. We need to clean 
some things up on the projectnessie Github org side first.


Robert

On 20.02.25 09:57, Jean-Baptiste Onofré wrote:

Hi Dmitri

About "evolution plan", I see the catalog migrator tool evolving as a
set of beans/providers that will be used in both CLI, and some server
features (like federated catalogs or "foreign catalogs").
We should not focus too much on catalog migrator as it is today but
more how it will be tomorrow.

That's why I'm more in favor of preparing the field and donating as a
module in the Polaris repo.

Regards
JB

On Thu, Feb 20, 2025 at 6:38 AM Dmitri Bourlatchkov  wrote:

+1 to accept the catalog migrator tool.

I support inviting Ajantha as a committer.

As to the source location, I tend to think that a separate repo makes sense
with the current state of the code, but I also agree that the overhead of
that may be too much, given that the codebase is small. I'm fine with
either a separate repo or a new module in the current Polaris repo.

What is the general plan for the evolution of the migrator tool? Are we
talking about integrating it into Polaris Servers or will it remain a
standalone tool as it is now?

Thanks,
Dmitri.

On Wed, Feb 19, 2025 at 11:39 AM Jean-Baptiste Onofré 
wrote:


Hi folks,

Let me try to sum-up this topic.

1. Catalog Migration landing
It seems we have a preference to land catalog-migrator as a module on
the main polaris repo.
Robert expressed comments about CI, release cycle, dependencies.

My view on that is that the purpose of the catalog-migrator is to
evolve, and could become a key component for features like federated
catalogs.
Due to that, I think we can consider catalog-migrator as a module,
integrated in the Polaris CLI, or in the Polaris server,

Robert, does it work for you ?

2. Code/PR prep
I propose to work directly with Ajantha (main contributor of the
catalog-migrator) to prepare the code heading to a PR. We need:
- integrate in Polaris repo and gradle
- rename all packages to use org.apache.polaris
- add ASF header in all files
- refactore cli to use polaris style/naming
- refactore intTest to use Polaris instead of Nessie
- check the dependencies in the cli uber jar (hadoop, hive, ...) and
cleanup LICENSE/NOTICE there
- update README and cleanup other files
It should be pretty fast and we should be able to create a PR for
review/donation.

Thoughts ?

Regards
JB

On Tue, Feb 11, 2025 at 7:45 PM Jean-Baptiste Onofré 
wrote:

Hi folks,

As discussed with some of you, we would like to propose donating the
Nessie Iceberg Catalog migrator tool
(https://github.com/projectnessie/iceberg-catalog-migrator) to Apache
Polaris.

A SGA has been already signed in case we accept the donation.

In terms of donation, I propose the following:
- the iceberg-catalog-migration can land in a separate Polaris repo
(as it is today for Nessie):
https://github.com/apache/polaris-catalog-migration or land as a
module in polaris repo directly
- we need to prepare the donation by changing the package names, etc
- I would suggest to consider inviting one of the main contributor of
Catalog Migrator (ajantha-bhat) as Polaris committer

WDYT ? Do we accept the iceberg-catalog-migrator tool in Polaris (we
can do a formal vote if we don't have obvious consensus) ?

Regards
JB


--
Robert Stupp
@snazy