Hey Dmitri and Robert,
*To clarify our use case further:* This isn't a one-time migration for us. We're migrating our *customers* from Polaris 0.9 (EclipseLink) to Polaris 1.0 (JDBC) gradually. During this process, we’ll be *running both catalog servers in parallel*, with 1.0 acting as a *secondary/fallback* catalog. Our strategy involves: - Registering the same tables in both 0.9 and 1.0 - Using the *same *clientId* and *clientSecret in both catalogs to ensure clients can authenticate seamlessly - Allowing us to *switch traffic between the two catalogs*, and roll back instantly if needed This setup requires credential continuity — not just for migration, but to enable *safe rollback and zero-downtime cutover*. Using different credentials across catalogs versions would break this flow and require deep client coordination to rotate secrets, which is not feasible at scale. *Regarding your question, Dmitri: *I wonder how your tooling could obtain Principals' secrets from the old Polaris instance for use as the new Principal creation request parameter - We store the credentials in an external Vault as well. So we are not reading them from the old Polaris instance, but do have access to them. We did consider raw table copying, but the differences in schema and hashing logic between 0.9 and 1.0 make that risky — and harder to validate/test it completely due to unknown risks. So our goal with this proposal is to: - Enable a *safe, service-admin only way* to inject known credentials via the API during the transition phase with validations of course - Keep this functionality configurable. We’re not trying to expand Polaris into a full IdP — just to provide a secure and practical bridge between versions. So the change seems fine to us. Happy to iterate on the proposal in a future sync On Wed, Jul 9, 2025 at 3:49 AM Dmitri Bourlatchkov <di...@apache.org> wrote: > Hi Arun, > > Thank you for starting this discussion! > > I did some poking about Keycloak and it looks like Keycloak allows > user-provided Client IDs. > > I think it should be fine for Polaris to accept user-provided Client IDs in > the Principal management API. I suppose we may want to impose some > restrictions in terms of special characters, but in general a > previous Polaris Client ID should be valid as an input parameter when > creating a new Principal. > > I think it should also be fine for Polaris to accept user-provided Client > Secrets (passwords) when creating Principals. > > That said, from my POV using the Admin Tool is still preferable for > migration use cases. > > My main argument in favour of the Admin Tool is that the whole migration > process is a deployment type of activity when the Polaris service is > configured for the first time. Ideally, Polaris data would follow a backup > / restore process (not currently implemented) where the old instance's data > is exported into a file, which is then imported into the new instance > before it is started for the first time. > > I wonder how your tooling could obtain Principals' secrets from the old > Polaris instance for use as the new Principal creation request parameter. > > Cheers, > Dmitri. > > On Tue, Jul 8, 2025 at 9:02 AM Arun Suri <arun.s...@fivetran.com.invalid> > wrote: > > > Hi all, > > > > Following up on the suggestion from the discussion here > > <https://github.com/apache/polaris/issues/1929#issuecomment-3045487786> > > — thank you for the feedback so far. > > > > We’re currently migrating our self-hosted Polaris service from version > 0.9 > > (EclipseLink-based metastore) to version 1.0 (JDBC-based metastore). As > > part of this transition, we need to preserve the existing `clientId` and > > `clientSecret` credentials for registered principals. > > These credentials are already embedded in customer workflows. Rotating > them > > during migration would create disruptions and require cross-team > > coordination with clients — making both rollout and rollback > significantly > > more complex. > > > > We understand the security implications of allowing arbitrary credentials > > to be passed in an API request. That said, we believe this capability can > > be introduced safely and in a tightly controlled manner. For example: > > > > - Restricting this functionality to service admin only. > > - Ensuring all credential transmissions occur only over HTTPS > > - Clearly documenting that this is strictly for *migration/bootstrap use > > cases*, not for production use > > - Disabling this functionality by default in publicly hosted deployments > > - Ensuring credentials are never logged (e.g., in observability systems > > like logs or traces) > > > > Our goal is not to weaken the system's security guarantees, but to > provide > > a practical and secure migration path where credential continuity is > > essential. Since there are a lot of DB schema changes involved, Manual > > insertion into the metastore isn't ideal either, due to potential > > inconsistencies in hashing or salting logic across versions — increasing > > operational risk. > > > > *### Proposal* > > > > We propose extending the existing `createPrincipal` API to optionally > > accept `clientId` and `clientSecret` fields, with the above safeguards in > > place. > > > > We would appreciate your feedback on this proposal and are happy to > > contribute a patch once there’s alignment. We’re also open to discussing > > this during the next Polaris Community Sync if helpful. > > > > Arun Suri > > > > Senior Software Engineer > > > > He/him/his > > > > Engineering | Fivetran > > arun.s...@fivetran.com > > fivetran.com <//fivetran.com> > > <http://www.fivetran.com> > > [image: facebook] <https://www.facebook.com/Fivetran/> [image: twitter] > > < > > > https://twitter.com/fivetran?ref_src=twsrc%5Egoogle%7Ctwcamp%5Eserp%7Ctwgr%5Eauthor > > > > > [image: > > linkedin] <https://www.linkedin.com/company/fivetran> [image: instagram] > > <https://www.instagram.com/fivetran_ig/> > > > -- Arun Suri Senior Software Engineer He/him/his Engineering | Fivetran arun.s...@fivetran.com fivetran.com <//fivetran.com> <http://www.fivetran.com> [image: facebook] <https://www.facebook.com/Fivetran/> [image: twitter] <https://twitter.com/fivetran?ref_src=twsrc%5Egoogle%7Ctwcamp%5Eserp%7Ctwgr%5Eauthor> [image: linkedin] <https://www.linkedin.com/company/fivetran> [image: instagram] <https://www.instagram.com/fivetran_ig/>
