Hi folks, I created the https://github.com/apache/polaris-catalogs-migrator repository.
I will work with Ajantha to populate it. Regards JB On Thu, Feb 20, 2025 at 12:33 PM Jean-Baptiste Onofré <j...@nanthrax.net> wrote: > > OK, let's move forward then > > I will prepare https://github.com/apache/polaris-catalogs-migrator > repository as a transition phase. > > > I also support inviting Ajantha as a committer. > That will be a separate discussion with PPMC. > > > Still really prefer a separate repository, at least for now. We can always > > merge later. It also makes the "migration to ASF" easier. > > > > The "Nessie Iceberg-catalog-migrator tool" has been built to support > > migrations from any catalog to any other catalog. This means, that the test > > matrix is quite complex and I expect it to become even more complex and > > time consuming. IMHO there's no need to "bother" Polaris "main CI" for > > every PR with catalog-migrator tests. I would also expect a different > > release cycle - no need to tie both together. > > Understood, let's use a specific repo for now. > > > Better use org.apache.polaris.catalogmigrator as the base group ID. > > Yes, that was the intent indeed. > > > This requires https://github.com/apache/polaris/pull/785. I'd also prefer > > to keep the ITs against Nessie and add Polaris. > > The itests will probably evolve a lot depending of the new use of > catalogs-migrator. > The itests bring a bunch of dependencies (Nessie, Hive, ...). As we > are talking about test dependencies, that's OK. > > > > > We don't know yet how these (and other features) will look like and not how > > it'll be related to the donated tool. > > > > I'm very enthusiastic about catalogs-migrator: I see a lot of potential :) > For instance, the first "obvious" move is probably catalogs-migrator > will evolve from a "standalone tool" to a library that we can use in > Polaris server (federated catalog, etc). > > > > > Please coordinate the migration from the source repo with me. We need to > > clean some things up on the projectnessie Github org side first. > > > > Ack, I will ping you :) > > For the rest of the community, no objection to starting with > https://github.com/apache/polaris-catalogs-migrator ? > > Regards > JB > > > > > Robert > > > > On 20.02.25 09:57, Jean-Baptiste Onofré wrote: > > > Hi Dmitri > > > > > > About "evolution plan", I see the catalog migrator tool evolving as a > > > set of beans/providers that will be used in both CLI, and some server > > > features (like federated catalogs or "foreign catalogs"). > > > We should not focus too much on catalog migrator as it is today but > > > more how it will be tomorrow. > > > > > > That's why I'm more in favor of preparing the field and donating as a > > > module in the Polaris repo. > > > > > > Regards > > > JB > > > > > > On Thu, Feb 20, 2025 at 6:38 AM Dmitri Bourlatchkov <di...@apache.org> > > > wrote: > > >> +1 to accept the catalog migrator tool. > > >> > > >> I support inviting Ajantha as a committer. > > >> > > >> As to the source location, I tend to think that a separate repo makes > > >> sense > > >> with the current state of the code, but I also agree that the overhead of > > >> that may be too much, given that the codebase is small. I'm fine with > > >> either a separate repo or a new module in the current Polaris repo. > > >> > > >> What is the general plan for the evolution of the migrator tool? Are we > > >> talking about integrating it into Polaris Servers or will it remain a > > >> standalone tool as it is now? > > >> > > >> Thanks, > > >> Dmitri. > > >> > > >> On Wed, Feb 19, 2025 at 11:39 AM Jean-Baptiste Onofré <j...@nanthrax.net> > > >> wrote: > > >> > > >>> Hi folks, > > >>> > > >>> Let me try to sum-up this topic. > > >>> > > >>> 1. Catalog Migration landing > > >>> It seems we have a preference to land catalog-migrator as a module on > > >>> the main polaris repo. > > >>> Robert expressed comments about CI, release cycle, dependencies. > > >>> > > >>> My view on that is that the purpose of the catalog-migrator is to > > >>> evolve, and could become a key component for features like federated > > >>> catalogs. > > >>> Due to that, I think we can consider catalog-migrator as a module, > > >>> integrated in the Polaris CLI, or in the Polaris server, > > >>> > > >>> Robert, does it work for you ? > > >>> > > >>> 2. Code/PR prep > > >>> I propose to work directly with Ajantha (main contributor of the > > >>> catalog-migrator) to prepare the code heading to a PR. We need: > > >>> - integrate in Polaris repo and gradle > > >>> - rename all packages to use org.apache.polaris > > >>> - add ASF header in all files > > >>> - refactore cli to use polaris style/naming > > >>> - refactore intTest to use Polaris instead of Nessie > > >>> - check the dependencies in the cli uber jar (hadoop, hive, ...) and > > >>> cleanup LICENSE/NOTICE there > > >>> - update README and cleanup other files > > >>> It should be pretty fast and we should be able to create a PR for > > >>> review/donation. > > >>> > > >>> Thoughts ? > > >>> > > >>> Regards > > >>> JB > > >>> > > >>> On Tue, Feb 11, 2025 at 7:45 PM Jean-Baptiste Onofré <j...@nanthrax.net> > > >>> wrote: > > >>>> Hi folks, > > >>>> > > >>>> As discussed with some of you, we would like to propose donating the > > >>>> Nessie Iceberg Catalog migrator tool > > >>>> (https://github.com/projectnessie/iceberg-catalog-migrator) to Apache > > >>>> Polaris. > > >>>> > > >>>> A SGA has been already signed in case we accept the donation. > > >>>> > > >>>> In terms of donation, I propose the following: > > >>>> - the iceberg-catalog-migration can land in a separate Polaris repo > > >>>> (as it is today for Nessie): > > >>>> https://github.com/apache/polaris-catalog-migration or land as a > > >>>> module in polaris repo directly > > >>>> - we need to prepare the donation by changing the package names, etc > > >>>> - I would suggest to consider inviting one of the main contributor of > > >>>> Catalog Migrator (ajantha-bhat) as Polaris committer > > >>>> > > >>>> WDYT ? Do we accept the iceberg-catalog-migrator tool in Polaris (we > > >>>> can do a formal vote if we don't have obvious consensus) ? > > >>>> > > >>>> Regards > > >>>> JB > > > > -- > > Robert Stupp > > @snazy > >
