Hi Arun, After writing my previous email I realised that we might be able to support your use case via REST API by implementing admin-base password reset [624].
The flow would be: 1) create a principal using current ID (random client ID/secret) 2) set your own client ID / secret via the password reset API (new) Would this work from your POV? As for me, it helps to isolate rare admin-level principal manipulation from the mainstream principal API, which IMHO, is a good idea from a general security perspective. Also, since we allow user-provided client IDs (after this change), it would make sense to allow resetting them without re-creating the principal, hence the new "reset" API. [624] https://github.com/apache/polaris/issues/624 Cheers, Dmitri. On Mon, Jul 14, 2025 at 11:29 AM Dmitri Bourlatchkov <di...@apache.org> wrote: > Thanks for volunteering to make a PR for this, Arun! Looking forward to it. > > As discussed, please consider securing the extra capability via (new) > permission checks. I'd think it might be worth it to also have a feature > flag to control the new functionality. > > Re: External IdP - most of the authentication code is already in `main`. > There are a few remaining dangling pieces related to connecting external > users to Polaris roles, though, IIRC. > > Cheers, > Dmitri. > > On Mon, Jul 14, 2025 at 6:34 AM Arun Suri <arun.s...@fivetran.com.invalid> > wrote: > >> Thanks for the detailed response Dimitri and Yufei! >> >> I agree with making the PR to support user-defined client ID and secret >> via >> the REST API, along with appropriate access checks and possibly >> introducing >> a new permission type/config. I will work on this >> >> REST fits better with our tooling as it has fewer dependencies and >> complications compared to the JAR-based Admin CLI. We also believe >> building >> our migration logic in a neutral way (e.g., using the REST API) is more >> robust—no matter how the tools evolve, the API remains the stable >> contract. >> >> As for external IdP delegation, it's something we're open to exploring >> down >> the line, though we understand it's still relatively new in Polaris >> >> Thanks, >> >> On Sun, Jul 13, 2025 at 1:28 AM Yufei Gu <flyrain...@gmail.com> wrote: >> >> > The use case of passing secrets via REST is definitely valid, and this >> use >> > case is not the only one we should be considering. Other concrete >> scenarios >> > include: >> > >> > 1. Catalog federation, where Polaris needs to store credentials to >> > connect to remote catalogs (e.g., Hive, Glue, Unity Catalog). >> > 2. S3-compatible storage without STS support, where Polaris must >> persist >> > static access keys and secrets to enable read/write operations. >> > >> > Given these needs, I think it's the right time to formalize our >> approach to >> > secret management by integrating Polaris with established secret >> managers >> > such as HashiCorp Vault, AWS Secrets Manager, Azure Key Vault, and >> Google >> > Cloud Secret Manager. >> > >> > While using the admin tool to inject secrets is a workable short-term >> > solution, it’s best treated as a stopgap. >> > >> > The good news is that the secret management interface was introduced in >> the >> > Polari core already, >> > >> > >> https://github.com/apache/polaris/blob/main/polaris-core/src/main/java/org/apache/polaris/core/secrets/UserSecretsManager.java >> > , >> > we may just need to provide wrapper implementations for different secret >> > managers. >> > >> > >> > Yufei >> > >> > >> > On Thu, Jul 10, 2025 at 12:52 PM Dmitri Bourlatchkov <di...@apache.org> >> > wrote: >> > >> > > Thanks for providing more context, Arun! >> > > >> > > I do not object to adding user-provided client ID and secret to the >> REST >> > > API. However, I personally maintain my opinion that this kind of >> > operation >> > > fits better with the Admin Tool, given the current state of the >> project. >> > I >> > > wonder what other community members think on this topic, too. >> > > >> > > If we go with updating the current REST API, then limiting access to >> > > explicit client ID and secret parameters via access checks will >> certainly >> > > make sense. We may need a new permission type for this, I guess. >> > > >> > > Do you have the capacity to make a PR for this? >> > > >> > > Regarding the Admin Tool, is the difficulty in the fact that it is a >> CLI >> > > tool that requires a JVM and your existing tooling is based on >> HTTP/REST >> > > and is not written in java? Just trying to understand the overall use >> > case >> > > better. >> > > >> > > Your point about the external vault makes me wonder whether you might >> be >> > > interested in running an IdP server (e.g. keycloak) in your infra and >> > > making Polaris delegate user management to that system. There's some >> > > existing support for that, but I'm not sure if anyone tried it >> end-to-end >> > > without any custom code on the server side (it is certainly possible >> with >> > > custom code). >> > > >> > > Thanks, >> > > Dmitri. >> > > >> > > On Thu, Jul 10, 2025 at 3:30 PM Arun Suri <arun.s...@fivetran.com >> > .invalid> >> > > wrote: >> > > >> > > > Hey Dmitri and Robert, >> > > > >> > > > >> > > > *To clarify our use case further:* >> > > > >> > > > >> > > > This isn't a one-time migration for us. We're migrating our >> *customers* >> > > > from >> > > > Polaris 0.9 (EclipseLink) to Polaris 1.0 (JDBC) gradually. During >> this >> > > > process, we’ll be *running both catalog servers in parallel*, with >> 1.0 >> > > > acting as a *secondary/fallback* catalog. >> > > > >> > > > Our strategy involves: >> > > > >> > > > - Registering the same tables in both 0.9 and 1.0 >> > > > - Using the *same *clientId* and *clientSecret in both catalogs >> to >> > > > ensure clients can authenticate seamlessly >> > > > - Allowing us to *switch traffic between the two catalogs*, and >> roll >> > > > back instantly if needed >> > > > >> > > > This setup requires credential continuity — not just for migration, >> but >> > > to >> > > > enable *safe rollback and zero-downtime cutover*. Using different >> > > > credentials across catalogs versions would break this flow and >> require >> > > deep >> > > > client coordination to rotate secrets, which is not feasible at >> scale. >> > > > >> > > > >> > > > *Regarding your question, Dmitri: *I wonder how your tooling could >> > obtain >> > > > Principals' secrets from the old >> > > > >> > > > Polaris instance for use as the new Principal creation request >> > parameter >> > > > >> > > > - We store the credentials in an external Vault as well. So we are >> not >> > > > reading them from the old Polaris instance, but do have access to >> them. >> > > > >> > > > >> > > > We did consider raw table copying, but the differences in schema and >> > > > hashing logic between 0.9 and 1.0 make that risky — and harder to >> > > > validate/test it completely due to unknown risks. >> > > > >> > > > So our goal with this proposal is to: >> > > > >> > > > - Enable a *safe, service-admin only way* to inject known >> > credentials >> > > > via the API during the transition phase with validations of >> course >> > > > - Keep this functionality configurable. >> > > > >> > > > We’re not trying to expand Polaris into a full IdP — just to >> provide a >> > > > secure and practical bridge between versions. So the change seems >> fine >> > to >> > > > us. >> > > > >> > > > Happy to iterate on the proposal in a future sync >> > > > >> > > > On Wed, Jul 9, 2025 at 3:49 AM Dmitri Bourlatchkov < >> di...@apache.org> >> > > > wrote: >> > > > >> > > > > Hi Arun, >> > > > > >> > > > > Thank you for starting this discussion! >> > > > > >> > > > > I did some poking about Keycloak and it looks like Keycloak allows >> > > > > user-provided Client IDs. >> > > > > >> > > > > I think it should be fine for Polaris to accept user-provided >> Client >> > > IDs >> > > > in >> > > > > the Principal management API. I suppose we may want to impose some >> > > > > restrictions in terms of special characters, but in general a >> > > > > previous Polaris Client ID should be valid as an input parameter >> when >> > > > > creating a new Principal. >> > > > > >> > > > > I think it should also be fine for Polaris to accept user-provided >> > > Client >> > > > > Secrets (passwords) when creating Principals. >> > > > > >> > > > > That said, from my POV using the Admin Tool is still preferable >> for >> > > > > migration use cases. >> > > > > >> > > > > My main argument in favour of the Admin Tool is that the whole >> > > migration >> > > > > process is a deployment type of activity when the Polaris service >> is >> > > > > configured for the first time. Ideally, Polaris data would follow >> a >> > > > backup >> > > > > / restore process (not currently implemented) where the old >> > instance's >> > > > data >> > > > > is exported into a file, which is then imported into the new >> instance >> > > > > before it is started for the first time. >> > > > > >> > > > > I wonder how your tooling could obtain Principals' secrets from >> the >> > old >> > > > > Polaris instance for use as the new Principal creation request >> > > parameter. >> > > > > >> > > > > Cheers, >> > > > > Dmitri. >> > > > > >> > > > > On Tue, Jul 8, 2025 at 9:02 AM Arun Suri <arun.s...@fivetran.com >> > > > .invalid> >> > > > > wrote: >> > > > > >> > > > > > Hi all, >> > > > > > >> > > > > > Following up on the suggestion from the discussion here >> > > > > > < >> > > https://github.com/apache/polaris/issues/1929#issuecomment-3045487786 >> > > > > >> > > > > > — thank you for the feedback so far. >> > > > > > >> > > > > > We’re currently migrating our self-hosted Polaris service from >> > > version >> > > > > 0.9 >> > > > > > (EclipseLink-based metastore) to version 1.0 (JDBC-based >> > metastore). >> > > As >> > > > > > part of this transition, we need to preserve the existing >> > `clientId` >> > > > and >> > > > > > `clientSecret` credentials for registered principals. >> > > > > > These credentials are already embedded in customer workflows. >> > > Rotating >> > > > > them >> > > > > > during migration would create disruptions and require cross-team >> > > > > > coordination with clients — making both rollout and rollback >> > > > > significantly >> > > > > > more complex. >> > > > > > >> > > > > > We understand the security implications of allowing arbitrary >> > > > credentials >> > > > > > to be passed in an API request. That said, we believe this >> > capability >> > > > can >> > > > > > be introduced safely and in a tightly controlled manner. For >> > example: >> > > > > > >> > > > > > - Restricting this functionality to service admin only. >> > > > > > - Ensuring all credential transmissions occur only over HTTPS >> > > > > > - Clearly documenting that this is strictly for >> > *migration/bootstrap >> > > > use >> > > > > > cases*, not for production use >> > > > > > - Disabling this functionality by default in publicly hosted >> > > > deployments >> > > > > > - Ensuring credentials are never logged (e.g., in observability >> > > systems >> > > > > > like logs or traces) >> > > > > > >> > > > > > Our goal is not to weaken the system's security guarantees, but >> to >> > > > > provide >> > > > > > a practical and secure migration path where credential >> continuity >> > is >> > > > > > essential. Since there are a lot of DB schema changes involved, >> > > Manual >> > > > > > insertion into the metastore isn't ideal either, due to >> potential >> > > > > > inconsistencies in hashing or salting logic across versions — >> > > > increasing >> > > > > > operational risk. >> > > > > > >> > > > > > *### Proposal* >> > > > > > >> > > > > > We propose extending the existing `createPrincipal` API to >> > optionally >> > > > > > accept `clientId` and `clientSecret` fields, with the above >> > > safeguards >> > > > in >> > > > > > place. >> > > > > > >> > > > > > We would appreciate your feedback on this proposal and are >> happy to >> > > > > > contribute a patch once there’s alignment. We’re also open to >> > > > discussing >> > > > > > this during the next Polaris Community Sync if helpful. >> > > > > > >> > > > > > Arun Suri >> > > > > > >> > > > > > Senior Software Engineer >> > > > > > >> > > > > > He/him/his >> > > > > > >> > > > > > Engineering | Fivetran >> > > > > > arun.s...@fivetran.com >> > > > > > fivetran.com <//fivetran.com> >> > > > > > <http://www.fivetran.com> >> > > > > > [image: facebook] <https://www.facebook.com/Fivetran/> [image: >> > > > twitter] >> > > > > > < >> > > > > > >> > > > > >> > > > >> > > >> > >> https://twitter.com/fivetran?ref_src=twsrc%5Egoogle%7Ctwcamp%5Eserp%7Ctwgr%5Eauthor >> > > > > > > >> > > > > > [image: >> > > > > > linkedin] <https://www.linkedin.com/company/fivetran> [image: >> > > > instagram] >> > > > > > <https://www.instagram.com/fivetran_ig/> >> > > > > > >> > > > > >> > > > >> > > > >> > > > -- >> > > > Arun Suri >> > > > >> > > > Senior Software Engineer >> > > > >> > > > He/him/his >> > > > >> > > > Engineering | Fivetran >> > > > arun.s...@fivetran.com >> > > > fivetran.com <//fivetran.com> >> > > > <http://www.fivetran.com> >> > > > [image: facebook] <https://www.facebook.com/Fivetran/> [image: >> > twitter] >> > > > < >> > > > >> > > >> > >> https://twitter.com/fivetran?ref_src=twsrc%5Egoogle%7Ctwcamp%5Eserp%7Ctwgr%5Eauthor >> > > > > >> > > > [image: >> > > > linkedin] <https://www.linkedin.com/company/fivetran> [image: >> > instagram] >> > > > <https://www.instagram.com/fivetran_ig/> >> > > > >> > > >> > >> >> >> -- >> Arun Suri >> >> Senior Software Engineer >> >> He/him/his >> >> Engineering | Fivetran >> arun.s...@fivetran.com >> fivetran.com <//fivetran.com> >> <http://www.fivetran.com> >> [image: facebook] <https://www.facebook.com/Fivetran/> [image: twitter] >> < >> https://twitter.com/fivetran?ref_src=twsrc%5Egoogle%7Ctwcamp%5Eserp%7Ctwgr%5Eauthor >> > >> [image: >> linkedin] <https://www.linkedin.com/company/fivetran> [image: instagram] >> <https://www.instagram.com/fivetran_ig/> >> >
