Re: [VOTE] Use renovatebot weekly schedule
On 24.02.25 10:54, Jean-Baptiste Onofré wrote: Hi Robert, As your vote is binding, and this vote is a code modification change, it basically means veto. Let me try to convince you to revert your vote ;) You are maybe right about the noise, but worth a try. I think that grouping + weekly schedule should reduce the noise. I'm confused now. This one is about "weekly schedule" but ARAIK we all agreed that "grouping everything" is definitely not an option, no? About keeping dependencies up-to-date, especially for bugs and security issues, I'm with you on that. That said, we can always have a "quick" update, so we can have weekly schedules and "on-demand" updates when needed (CVE, ...). Thoughts ? Currently it's sadly maybe 3 people who constantly tackle dependency updates. To be honest, I'm quite disappointed that not more people are interested in keeping dependencies up to date. Keeping those up-to-date _actively_ prevents tech debt from piling up and running into (already fixed) bugs and security issues. What was probably not considered at all: Weekly updates will _increase_ the amount of "noise". Just think about it: Renovate creates all updates at once - once one change is merged, Renovate will rebase/recreate all other updates, CI will run again for all the other updates. If all people are fine with more noise, let's go for it - I'd be happy to revert my vote if that's the case. Regards JB On Sun, Feb 23, 2025 at 1:04 PM Robert Stupp wrote: Keeping dependencies up-to-date is mandatory to get bug and security fixes as fast as possible and avoid piling up tech debt and CVEs from outdated dependencies. Moving to a weekly schedule would _not_ reduce that noise but instead _increase_ it and imply _more_ work to the few persons who deal with dependency updates. It is _much_ easier to deal with dependency updates as they are created. Therefore my (binding) -1 vote on this. On 21.02.25 14:38, Jean-Baptiste Onofré wrote: Hi folks, I know it's a hot topic, but I would like to avoid any frustration in our community. Before the vote, let me put some context. To manage our dependency updates, we are using renovatebot. The current renovatebot configuration uses "at any time" schedule (e.g. * * * * * cron), except for AWS SDK and boto3 updates which run weekly. Some contributors are complaining about the "noise" generated by renovatebot. In order to "mitigate" that, we introduced "polaris-renovate" label to easily filter the notification coming from renovatebot. However, an issue has been created 4 days ago (https://github.com/apache/polaris/issues/1018), meaning the "issue" is still there. So, I propose this vote to have clear feedback from the community, as we don't have clear lazy consensus. The vote is to schedule renovatebot update weekly: [ ] +1 - Use weekly schedule for all renovatebot updates [ ] 0 [ ] -1 - Don't use weekly schedule, keep the "at any time" schedule Thanks, Regards JB NB: we can consider this vote as a code modification vote (see https://www.apache.org/foundation/voting.html#votes-on-code-modification). -- Robert Stupp @snazy -- Robert Stupp @snazy
Re: Donate Nessie Iceberg Catalog migrator
I'm not sure if we want to limit it to Iceberg only as Polaris starts to support generic tables. I guess "catalog-migrator" is short and expressive. "polaris-catalog-migrator" is also fine to me, given that the package name already implies its association with Polaris. Yufei On Tue, Feb 25, 2025 at 5:25 PM Eric Maynard wrote: > iceberg-catalog-migrator sounds good to me! > > On Tue, Feb 25, 2025 at 5:23 PM Ajantha Bhat > wrote: > > > > > > > The Nessie migrator was `iceberg-catalog-migrator`, and "catalogs" does > > > not seem grammatical to me. If I'm grading papers, I'm not a papers > > grader, > > > I'm a paper grader. > > > > > > The original name, *iceberg-catalog-migrator*, suggests that it supports > > migrations between any Iceberg catalogs. Since we are placing it under > > Apache Polaris, we wanted to incorporate "Polaris" into the name. > However, > > naming it *polaris-catalog-migrator* might imply that it is exclusively > for > > migrating tables to or from the Polaris catalog. > > > > We need a better name or should consider keeping just > > *apache/iceberg-catalog-migrator* > > Package names in the source can be `org.apache.polaris.catalog.migrator` > > > > Suggestions are welcome! > > > > - Ajantha > > > > On Wed, Feb 26, 2025 at 1:20 AM Dmitri Bourlatchkov > > wrote: > > > > > Good point re: grammar, and I'll defer to native speakers on that :) > > > > > > On Tue, Feb 25, 2025 at 2:45 PM Eric Maynard > > > > wrote: > > > > > > > The Nessie migrator was `iceberg-catalog-migrator`, and "catalogs" > does > > > > not seem grammatical to me. If I'm grading papers, I'm not a papers > > > grader, > > > > I'm a paper grader. > > > > > > > > The fact that it works with different catalog implementations doesn't > > > > change that, but if we think polaris-catalog-migrator sounds too must > > > like > > > > it only can migrate to/from Polaris catalogs I bet we can come up > with > > a > > > > better name indeed. > > > > > > > > On Tue, Feb 25, 2025 at 11:35 AM Dmitri Bourlatchkov > > > > wrote: > > > > > > > > > > Small question, why catalog*s*? > > > > > > > > > > > > > > > > > The tool can migrate any Iceberg catalog to any other Iceberg > > catalog. > > > > > > > > > > > > > > >
Re: Donate Nessie Iceberg Catalog migrator
iceberg-catalog-migrator sounds good to me! On Tue, Feb 25, 2025 at 5:23 PM Ajantha Bhat wrote: > > > > The Nessie migrator was `iceberg-catalog-migrator`, and "catalogs" does > > not seem grammatical to me. If I'm grading papers, I'm not a papers > grader, > > I'm a paper grader. > > > The original name, *iceberg-catalog-migrator*, suggests that it supports > migrations between any Iceberg catalogs. Since we are placing it under > Apache Polaris, we wanted to incorporate "Polaris" into the name. However, > naming it *polaris-catalog-migrator* might imply that it is exclusively for > migrating tables to or from the Polaris catalog. > > We need a better name or should consider keeping just > *apache/iceberg-catalog-migrator* > Package names in the source can be `org.apache.polaris.catalog.migrator` > > Suggestions are welcome! > > - Ajantha > > On Wed, Feb 26, 2025 at 1:20 AM Dmitri Bourlatchkov > wrote: > > > Good point re: grammar, and I'll defer to native speakers on that :) > > > > On Tue, Feb 25, 2025 at 2:45 PM Eric Maynard > > wrote: > > > > > The Nessie migrator was `iceberg-catalog-migrator`, and "catalogs" does > > > not seem grammatical to me. If I'm grading papers, I'm not a papers > > grader, > > > I'm a paper grader. > > > > > > The fact that it works with different catalog implementations doesn't > > > change that, but if we think polaris-catalog-migrator sounds too must > > like > > > it only can migrate to/from Polaris catalogs I bet we can come up with > a > > > better name indeed. > > > > > > On Tue, Feb 25, 2025 at 11:35 AM Dmitri Bourlatchkov > > > wrote: > > > > > > > > Small question, why catalog*s*? > > > > > > > > > > > > > > The tool can migrate any Iceberg catalog to any other Iceberg > catalog. > > > > > > > > > >
Re: Donate Nessie Iceberg Catalog migrator
> > The Nessie migrator was `iceberg-catalog-migrator`, and "catalogs" does > not seem grammatical to me. If I'm grading papers, I'm not a papers grader, > I'm a paper grader. The original name, *iceberg-catalog-migrator*, suggests that it supports migrations between any Iceberg catalogs. Since we are placing it under Apache Polaris, we wanted to incorporate "Polaris" into the name. However, naming it *polaris-catalog-migrator* might imply that it is exclusively for migrating tables to or from the Polaris catalog. We need a better name or should consider keeping just *apache/iceberg-catalog-migrator* Package names in the source can be `org.apache.polaris.catalog.migrator` Suggestions are welcome! - Ajantha On Wed, Feb 26, 2025 at 1:20 AM Dmitri Bourlatchkov wrote: > Good point re: grammar, and I'll defer to native speakers on that :) > > On Tue, Feb 25, 2025 at 2:45 PM Eric Maynard > wrote: > > > The Nessie migrator was `iceberg-catalog-migrator`, and "catalogs" does > > not seem grammatical to me. If I'm grading papers, I'm not a papers > grader, > > I'm a paper grader. > > > > The fact that it works with different catalog implementations doesn't > > change that, but if we think polaris-catalog-migrator sounds too must > like > > it only can migrate to/from Polaris catalogs I bet we can come up with a > > better name indeed. > > > > On Tue, Feb 25, 2025 at 11:35 AM Dmitri Bourlatchkov > > wrote: > > > > > > Small question, why catalog*s*? > > > > > > > > > > > The tool can migrate any Iceberg catalog to any other Iceberg catalog. > > > > > >
Re: Donate Nessie Iceberg Catalog migrator
Good point! catalog-migrator sounds good to me. On Tue, Feb 25, 2025 at 6:08 PM Yufei Gu wrote: > I'm not sure if we want to limit it to Iceberg only as Polaris starts to > support generic tables. I guess "catalog-migrator" is short and expressive. > "polaris-catalog-migrator" is also fine to me, given that the package name > already implies its association with Polaris. > > Yufei > > > On Tue, Feb 25, 2025 at 5:25 PM Eric Maynard > wrote: > > > iceberg-catalog-migrator sounds good to me! > > > > On Tue, Feb 25, 2025 at 5:23 PM Ajantha Bhat > > wrote: > > > > > > > > > > The Nessie migrator was `iceberg-catalog-migrator`, and "catalogs" > does > > > > not seem grammatical to me. If I'm grading papers, I'm not a papers > > > grader, > > > > I'm a paper grader. > > > > > > > > > The original name, *iceberg-catalog-migrator*, suggests that it > supports > > > migrations between any Iceberg catalogs. Since we are placing it under > > > Apache Polaris, we wanted to incorporate "Polaris" into the name. > > However, > > > naming it *polaris-catalog-migrator* might imply that it is exclusively > > for > > > migrating tables to or from the Polaris catalog. > > > > > > We need a better name or should consider keeping just > > > *apache/iceberg-catalog-migrator* > > > Package names in the source can be > `org.apache.polaris.catalog.migrator` > > > > > > Suggestions are welcome! > > > > > > - Ajantha > > > > > > On Wed, Feb 26, 2025 at 1:20 AM Dmitri Bourlatchkov > > > wrote: > > > > > > > Good point re: grammar, and I'll defer to native speakers on that :) > > > > > > > > On Tue, Feb 25, 2025 at 2:45 PM Eric Maynard < > eric.w.mayn...@gmail.com > > > > > > > wrote: > > > > > > > > > The Nessie migrator was `iceberg-catalog-migrator`, and "catalogs" > > does > > > > > not seem grammatical to me. If I'm grading papers, I'm not a papers > > > > grader, > > > > > I'm a paper grader. > > > > > > > > > > The fact that it works with different catalog implementations > doesn't > > > > > change that, but if we think polaris-catalog-migrator sounds too > must > > > > like > > > > > it only can migrate to/from Polaris catalogs I bet we can come up > > with > > > a > > > > > better name indeed. > > > > > > > > > > On Tue, Feb 25, 2025 at 11:35 AM Dmitri Bourlatchkov > > > > > wrote: > > > > > > > > > > > > Small question, why catalog*s*? > > > > > > > > > > > > > > > > > > > > The tool can migrate any Iceberg catalog to any other Iceberg > > > catalog. > > > > > > > > > > > > > > > > > > > > >
Re: Donate Nessie Iceberg Catalog migrator
JB mentioned that it has to have `polaris` in the repo name. How about calling it as `apache/*polaris-dev-tools*`? It can have an iceberg catalog migrator CLI, Delta lake catalog migrator CLI, or something else in the future as part of this repo. I don't want the repo to be called just `apache/polaris-catalog-migrator` because it sounds like it is just a migrator from/to polaris catalog. - Ajantha
Re: Donate Nessie Iceberg Catalog migrator
"polaris-tools" sounds good to me. I assume it's not only used by developers. Yufei On Tue, Feb 25, 2025 at 10:56 PM Ajantha Bhat wrote: > JB mentioned that it has to have `polaris` in the repo name. > > How about calling it as `apache/*polaris-dev-tools*`? It can have an > iceberg catalog migrator CLI, Delta lake catalog migrator CLI, or something > else in the future as part of this repo. > > I don't want the repo to be called just `apache/polaris-catalog-migrator` > because it sounds like it is just a migrator from/to polaris catalog. > > - Ajantha >
Re: Donate Nessie Iceberg Catalog migrator
> Small question, why catalog*s*? > > The tool can migrate any Iceberg catalog to any other Iceberg catalog.
[RESULT][VOTE] Release Apache Polaris (incubating) 0.9.0-rc6
Hi folks, This vote passed with the following result: +1 (binding): Ryan Blue, JB Onofré, PJ Fanning, Kent Yao Thanks everyone for your vote. I will finalize the release. Regards JB On Fri, Feb 21, 2025 at 6:57 AM Jean-Baptiste Onofré wrote: > > Hi folks, > > This is a call for the vote to release Apache Polaris (incubating) 0.9.0-rc6. > > The Polaris community has voted and approved Apache Polaris > (incubating) 0.9.0-rc6. > We are now asking the IPMC to review and vote for this release. > > Polaris community vote thread: > https://lists.apache.org/thread/7jxoswynzpnhsmbt77t4klc7rp983o66 > > Polaris community vote result: > https://lists.apache.org/thread/3ymqrtlvtcq1pgtd14ghn2n158y6yl5t > > * This corresponds to the tag: apache-polaris-0.9.0-incubating-rc6 > * https://github.com/apache/polaris/tree/apache-polaris-0.9.0-incubating-rc6 > * > https://github.com/apache/polaris/tree/4b18ec065ff16f74b11bc85fdc6ea9036eca7274 > > The release source tarball, signature, and checksums are here: > * https://dist.apache.org/repos/dist/dev/incubator/polaris/0.9.0-incubating/ > > You can find the KEYS file here: > * https://dist.apache.org/repos/dist/release/incubator/polaris/KEYS > > The vote will be open for at least 72 hours or until the necessary > number of votes are reached. > > Please vote accordingly: > > [ ] +1 Release this as Apache Polaris (incubating) 0.9.0 > [ ] +0 > [ ] -1 Do not release this because... > > This vote will pass if there are 3 binding +1 votes and more binding > +1 votes than -1 votes. > > Thanks, > Regards > JB
Re: Donate Nessie Iceberg Catalog migrator
Small question, why catalog*s*? On Tue, Feb 25, 2025 at 5:51 AM Jean-Baptiste Onofré wrote: > Hi folks, > > I created the https://github.com/apache/polaris-catalogs-migrator > repository. > > I will work with Ajantha to populate it. > > Regards > JB > > On Thu, Feb 20, 2025 at 12:33 PM Jean-Baptiste Onofré > wrote: > > > > OK, let's move forward then > > > > I will prepare https://github.com/apache/polaris-catalogs-migrator > > repository as a transition phase. > > > > > I also support inviting Ajantha as a committer. > > That will be a separate discussion with PPMC. > > > > > Still really prefer a separate repository, at least for now. We can > always merge later. It also makes the "migration to ASF" easier. > > > > > > The "Nessie Iceberg-catalog-migrator tool" has been built to support > migrations from any catalog to any other catalog. This means, that the test > matrix is quite complex and I expect it to become even more complex and > time consuming. IMHO there's no need to "bother" Polaris "main CI" for > every PR with catalog-migrator tests. I would also expect a different > release cycle - no need to tie both together. > > > > Understood, let's use a specific repo for now. > > > > > Better use org.apache.polaris.catalogmigrator as the base group ID. > > > > Yes, that was the intent indeed. > > > > > This requires https://github.com/apache/polaris/pull/785. I'd also > prefer to keep the ITs against Nessie and add Polaris. > > > > The itests will probably evolve a lot depending of the new use of > > catalogs-migrator. > > The itests bring a bunch of dependencies (Nessie, Hive, ...). As we > > are talking about test dependencies, that's OK. > > > > > > > > We don't know yet how these (and other features) will look like and > not how it'll be related to the donated tool. > > > > > > > I'm very enthusiastic about catalogs-migrator: I see a lot of potential > :) > > For instance, the first "obvious" move is probably catalogs-migrator > > will evolve from a "standalone tool" to a library that we can use in > > Polaris server (federated catalog, etc). > > > > > > > > Please coordinate the migration from the source repo with me. We need > to clean some things up on the projectnessie Github org side first. > > > > > > > Ack, I will ping you :) > > > > For the rest of the community, no objection to starting with > > https://github.com/apache/polaris-catalogs-migrator ? > > > > Regards > > JB > > > > > > > > Robert > > > > > > On 20.02.25 09:57, Jean-Baptiste Onofré wrote: > > > > Hi Dmitri > > > > > > > > About "evolution plan", I see the catalog migrator tool evolving as a > > > > set of beans/providers that will be used in both CLI, and some server > > > > features (like federated catalogs or "foreign catalogs"). > > > > We should not focus too much on catalog migrator as it is today but > > > > more how it will be tomorrow. > > > > > > > > That's why I'm more in favor of preparing the field and donating as a > > > > module in the Polaris repo. > > > > > > > > Regards > > > > JB > > > > > > > > On Thu, Feb 20, 2025 at 6:38 AM Dmitri Bourlatchkov < > di...@apache.org> wrote: > > > >> +1 to accept the catalog migrator tool. > > > >> > > > >> I support inviting Ajantha as a committer. > > > >> > > > >> As to the source location, I tend to think that a separate repo > makes sense > > > >> with the current state of the code, but I also agree that the > overhead of > > > >> that may be too much, given that the codebase is small. I'm fine > with > > > >> either a separate repo or a new module in the current Polaris repo. > > > >> > > > >> What is the general plan for the evolution of the migrator tool? > Are we > > > >> talking about integrating it into Polaris Servers or will it remain > a > > > >> standalone tool as it is now? > > > >> > > > >> Thanks, > > > >> Dmitri. > > > >> > > > >> On Wed, Feb 19, 2025 at 11:39 AM Jean-Baptiste Onofré < > j...@nanthrax.net> > > > >> wrote: > > > >> > > > >>> Hi folks, > > > >>> > > > >>> Let me try to sum-up this topic. > > > >>> > > > >>> 1. Catalog Migration landing > > > >>> It seems we have a preference to land catalog-migrator as a module > on > > > >>> the main polaris repo. > > > >>> Robert expressed comments about CI, release cycle, dependencies. > > > >>> > > > >>> My view on that is that the purpose of the catalog-migrator is to > > > >>> evolve, and could become a key component for features like > federated > > > >>> catalogs. > > > >>> Due to that, I think we can consider catalog-migrator as a module, > > > >>> integrated in the Polaris CLI, or in the Polaris server, > > > >>> > > > >>> Robert, does it work for you ? > > > >>> > > > >>> 2. Code/PR prep > > > >>> I propose to work directly with Ajantha (main contributor of the > > > >>> catalog-migrator) to prepare the code heading to a PR. We need: > > > >>> - integrate in Polaris repo and gradle > > > >>> - rename all packages to use org.apache.polaris > > > >>> - add ASF header in all files > > > >>> - ref
Re: Donate Nessie Iceberg Catalog migrator
The Nessie migrator was `iceberg-catalog-migrator`, and "catalogs" does not seem grammatical to me. If I'm grading papers, I'm not a papers grader, I'm a paper grader. The fact that it works with different catalog implementations doesn't change that, but if we think polaris-catalog-migrator sounds too must like it only can migrate to/from Polaris catalogs I bet we can come up with a better name indeed. On Tue, Feb 25, 2025 at 11:35 AM Dmitri Bourlatchkov wrote: > > Small question, why catalog*s*? > > > > > The tool can migrate any Iceberg catalog to any other Iceberg catalog. >
Re: Donate Nessie Iceberg Catalog migrator
Good point re: grammar, and I'll defer to native speakers on that :) On Tue, Feb 25, 2025 at 2:45 PM Eric Maynard wrote: > The Nessie migrator was `iceberg-catalog-migrator`, and "catalogs" does > not seem grammatical to me. If I'm grading papers, I'm not a papers grader, > I'm a paper grader. > > The fact that it works with different catalog implementations doesn't > change that, but if we think polaris-catalog-migrator sounds too must like > it only can migrate to/from Polaris catalogs I bet we can come up with a > better name indeed. > > On Tue, Feb 25, 2025 at 11:35 AM Dmitri Bourlatchkov > wrote: > > > > Small question, why catalog*s*? > > > > > > > > The tool can migrate any Iceberg catalog to any other Iceberg catalog. > > >
Re: [VOTE] Use renovatebot weekly schedule
> Weekly updates will _increase_ the amount of "noise" ... If all people are fine with more noise, let's go for it That does seem to be what the votes reflect. On Tue, Feb 25, 2025 at 7:11 AM Dmitri Bourlatchkov wrote: > What was probably not considered at all: Weekly updates will _increase_ > the amount of "noise". Just think about it: Renovate creates all updates > at once - once one change is merged, Renovate will rebase/recreate all > other updates, CI will run again for all the other updates. > > > Do we have to rebase Renovate PRs on every merge? > > Thanks, > Dmitri. >
Re: [VOTE] Use renovatebot weekly schedule
What was probably not considered at all: Weekly updates will _increase_ the amount of "noise". Just think about it: Renovate creates all updates at once - once one change is merged, Renovate will rebase/recreate all other updates, CI will run again for all the other updates. Do we have to rebase Renovate PRs on every merge? Thanks, Dmitri.
Re: [VOTE] Use renovatebot weekly schedule
The vote is only about weekly schedule (not grouping). I mentioned grouping to reduce the number of PRs but I’m more in favor of keeping atomic PRs. Regards JB Le mar. 25 févr. 2025 à 11:09, Robert Stupp a écrit : > > On 24.02.25 10:54, Jean-Baptiste Onofré wrote: > > Hi Robert, > > > > As your vote is binding, and this vote is a code modification change, > > it basically means veto. > > > > Let me try to convince you to revert your vote ;) > > You are maybe right about the noise, but worth a try. I think that > > grouping + weekly schedule should reduce the noise. > > I'm confused now. This one is about "weekly schedule" but ARAIK we all > agreed that "grouping everything" is definitely not an option, no? > > > About keeping dependencies up-to-date, especially for bugs and > > security issues, I'm with you on that. > > That said, we can always have a "quick" update, so we can have weekly > > schedules and "on-demand" updates when needed (CVE, ...). > > > > Thoughts ? > > Currently it's sadly maybe 3 people who constantly tackle dependency > updates. To be honest, I'm quite disappointed that not more people are > interested in keeping dependencies up to date. Keeping those up-to-date > _actively_ prevents tech debt from piling up and running into (already > fixed) bugs and security issues. > > What was probably not considered at all: Weekly updates will _increase_ > the amount of "noise". Just think about it: Renovate creates all updates > at once - once one change is merged, Renovate will rebase/recreate all > other updates, CI will run again for all the other updates. > > If all people are fine with more noise, let's go for it - I'd be happy > to revert my vote if that's the case. > > > Regards > > JB > > > > On Sun, Feb 23, 2025 at 1:04 PM Robert Stupp wrote: > >> Keeping dependencies up-to-date is mandatory to get bug and security > >> fixes as fast as possible and avoid piling up tech debt and CVEs from > >> outdated dependencies. > >> > >> Moving to a weekly schedule would _not_ reduce that noise but instead > >> _increase_ it and imply _more_ work to the few persons who deal with > >> dependency updates. > >> > >> It is _much_ easier to deal with dependency updates as they are created. > >> > >> Therefore my (binding) -1 vote on this. > >> > >> On 21.02.25 14:38, Jean-Baptiste Onofré wrote: > >>> Hi folks, > >>> > >>> I know it's a hot topic, but I would like to avoid any frustration in > >>> our community. > >>> > >>> Before the vote, let me put some context. > >>> > >>> To manage our dependency updates, we are using renovatebot. > >>> The current renovatebot configuration uses "at any time" schedule > >>> (e.g. * * * * * cron), except for AWS SDK and boto3 updates which run > >>> weekly. > >>> Some contributors are complaining about the "noise" generated by > renovatebot. > >>> In order to "mitigate" that, we introduced "polaris-renovate" label to > >>> easily filter the notification coming from renovatebot. > >>> However, an issue has been created 4 days ago > >>> (https://github.com/apache/polaris/issues/1018), meaning the "issue" > >>> is still there. > >>> > >>> So, I propose this vote to have clear feedback from the community, as > >>> we don't have clear lazy consensus. > >>> > >>> The vote is to schedule renovatebot update weekly: > >>> > >>> [ ] +1 - Use weekly schedule for all renovatebot updates > >>> [ ] 0 > >>> [ ] -1 - Don't use weekly schedule, keep the "at any time" schedule > >>> > >>> Thanks, > >>> Regards > >>> JB > >>> > >>> NB: we can consider this vote as a code modification vote (see > >>> > https://www.apache.org/foundation/voting.html#votes-on-code-modification). > >> -- > >> Robert Stupp > >> @snazy > >> > -- > Robert Stupp > @snazy > >
Re: Donate Nessie Iceberg Catalog migrator
Hi folks, I created the https://github.com/apache/polaris-catalogs-migrator repository. I will work with Ajantha to populate it. Regards JB On Thu, Feb 20, 2025 at 12:33 PM Jean-Baptiste Onofré wrote: > > OK, let's move forward then > > I will prepare https://github.com/apache/polaris-catalogs-migrator > repository as a transition phase. > > > I also support inviting Ajantha as a committer. > That will be a separate discussion with PPMC. > > > Still really prefer a separate repository, at least for now. We can always > > merge later. It also makes the "migration to ASF" easier. > > > > The "Nessie Iceberg-catalog-migrator tool" has been built to support > > migrations from any catalog to any other catalog. This means, that the test > > matrix is quite complex and I expect it to become even more complex and > > time consuming. IMHO there's no need to "bother" Polaris "main CI" for > > every PR with catalog-migrator tests. I would also expect a different > > release cycle - no need to tie both together. > > Understood, let's use a specific repo for now. > > > Better use org.apache.polaris.catalogmigrator as the base group ID. > > Yes, that was the intent indeed. > > > This requires https://github.com/apache/polaris/pull/785. I'd also prefer > > to keep the ITs against Nessie and add Polaris. > > The itests will probably evolve a lot depending of the new use of > catalogs-migrator. > The itests bring a bunch of dependencies (Nessie, Hive, ...). As we > are talking about test dependencies, that's OK. > > > > > We don't know yet how these (and other features) will look like and not how > > it'll be related to the donated tool. > > > > I'm very enthusiastic about catalogs-migrator: I see a lot of potential :) > For instance, the first "obvious" move is probably catalogs-migrator > will evolve from a "standalone tool" to a library that we can use in > Polaris server (federated catalog, etc). > > > > > Please coordinate the migration from the source repo with me. We need to > > clean some things up on the projectnessie Github org side first. > > > > Ack, I will ping you :) > > For the rest of the community, no objection to starting with > https://github.com/apache/polaris-catalogs-migrator ? > > Regards > JB > > > > > Robert > > > > On 20.02.25 09:57, Jean-Baptiste Onofré wrote: > > > Hi Dmitri > > > > > > About "evolution plan", I see the catalog migrator tool evolving as a > > > set of beans/providers that will be used in both CLI, and some server > > > features (like federated catalogs or "foreign catalogs"). > > > We should not focus too much on catalog migrator as it is today but > > > more how it will be tomorrow. > > > > > > That's why I'm more in favor of preparing the field and donating as a > > > module in the Polaris repo. > > > > > > Regards > > > JB > > > > > > On Thu, Feb 20, 2025 at 6:38 AM Dmitri Bourlatchkov > > > wrote: > > >> +1 to accept the catalog migrator tool. > > >> > > >> I support inviting Ajantha as a committer. > > >> > > >> As to the source location, I tend to think that a separate repo makes > > >> sense > > >> with the current state of the code, but I also agree that the overhead of > > >> that may be too much, given that the codebase is small. I'm fine with > > >> either a separate repo or a new module in the current Polaris repo. > > >> > > >> What is the general plan for the evolution of the migrator tool? Are we > > >> talking about integrating it into Polaris Servers or will it remain a > > >> standalone tool as it is now? > > >> > > >> Thanks, > > >> Dmitri. > > >> > > >> On Wed, Feb 19, 2025 at 11:39 AM Jean-Baptiste Onofré > > >> wrote: > > >> > > >>> Hi folks, > > >>> > > >>> Let me try to sum-up this topic. > > >>> > > >>> 1. Catalog Migration landing > > >>> It seems we have a preference to land catalog-migrator as a module on > > >>> the main polaris repo. > > >>> Robert expressed comments about CI, release cycle, dependencies. > > >>> > > >>> My view on that is that the purpose of the catalog-migrator is to > > >>> evolve, and could become a key component for features like federated > > >>> catalogs. > > >>> Due to that, I think we can consider catalog-migrator as a module, > > >>> integrated in the Polaris CLI, or in the Polaris server, > > >>> > > >>> Robert, does it work for you ? > > >>> > > >>> 2. Code/PR prep > > >>> I propose to work directly with Ajantha (main contributor of the > > >>> catalog-migrator) to prepare the code heading to a PR. We need: > > >>> - integrate in Polaris repo and gradle > > >>> - rename all packages to use org.apache.polaris > > >>> - add ASF header in all files > > >>> - refactore cli to use polaris style/naming > > >>> - refactore intTest to use Polaris instead of Nessie > > >>> - check the dependencies in the cli uber jar (hadoop, hive, ...) and > > >>> cleanup LICENSE/NOTICE there > > >>> - update README and cleanup other files > > >>> It should be pretty fast and we should be able to create a PR for > > >>> review
