The use case of passing secrets via REST is definitely valid, and this use case is not the only one we should be considering. Other concrete scenarios include:
1. Catalog federation, where Polaris needs to store credentials to connect to remote catalogs (e.g., Hive, Glue, Unity Catalog). 2. S3-compatible storage without STS support, where Polaris must persist static access keys and secrets to enable read/write operations. Given these needs, I think it's the right time to formalize our approach to secret management by integrating Polaris with established secret managers such as HashiCorp Vault, AWS Secrets Manager, Azure Key Vault, and Google Cloud Secret Manager. While using the admin tool to inject secrets is a workable short-term solution, it’s best treated as a stopgap. The good news is that the secret management interface was introduced in the Polari core already, https://github.com/apache/polaris/blob/main/polaris-core/src/main/java/org/apache/polaris/core/secrets/UserSecretsManager.java, we may just need to provide wrapper implementations for different secret managers. Yufei On Thu, Jul 10, 2025 at 12:52 PM Dmitri Bourlatchkov <di...@apache.org> wrote: > Thanks for providing more context, Arun! > > I do not object to adding user-provided client ID and secret to the REST > API. However, I personally maintain my opinion that this kind of operation > fits better with the Admin Tool, given the current state of the project. I > wonder what other community members think on this topic, too. > > If we go with updating the current REST API, then limiting access to > explicit client ID and secret parameters via access checks will certainly > make sense. We may need a new permission type for this, I guess. > > Do you have the capacity to make a PR for this? > > Regarding the Admin Tool, is the difficulty in the fact that it is a CLI > tool that requires a JVM and your existing tooling is based on HTTP/REST > and is not written in java? Just trying to understand the overall use case > better. > > Your point about the external vault makes me wonder whether you might be > interested in running an IdP server (e.g. keycloak) in your infra and > making Polaris delegate user management to that system. There's some > existing support for that, but I'm not sure if anyone tried it end-to-end > without any custom code on the server side (it is certainly possible with > custom code). > > Thanks, > Dmitri. > > On Thu, Jul 10, 2025 at 3:30 PM Arun Suri <arun.s...@fivetran.com.invalid> > wrote: > > > Hey Dmitri and Robert, > > > > > > *To clarify our use case further:* > > > > > > This isn't a one-time migration for us. We're migrating our *customers* > > from > > Polaris 0.9 (EclipseLink) to Polaris 1.0 (JDBC) gradually. During this > > process, we’ll be *running both catalog servers in parallel*, with 1.0 > > acting as a *secondary/fallback* catalog. > > > > Our strategy involves: > > > > - Registering the same tables in both 0.9 and 1.0 > > - Using the *same *clientId* and *clientSecret in both catalogs to > > ensure clients can authenticate seamlessly > > - Allowing us to *switch traffic between the two catalogs*, and roll > > back instantly if needed > > > > This setup requires credential continuity — not just for migration, but > to > > enable *safe rollback and zero-downtime cutover*. Using different > > credentials across catalogs versions would break this flow and require > deep > > client coordination to rotate secrets, which is not feasible at scale. > > > > > > *Regarding your question, Dmitri: *I wonder how your tooling could obtain > > Principals' secrets from the old > > > > Polaris instance for use as the new Principal creation request parameter > > > > - We store the credentials in an external Vault as well. So we are not > > reading them from the old Polaris instance, but do have access to them. > > > > > > We did consider raw table copying, but the differences in schema and > > hashing logic between 0.9 and 1.0 make that risky — and harder to > > validate/test it completely due to unknown risks. > > > > So our goal with this proposal is to: > > > > - Enable a *safe, service-admin only way* to inject known credentials > > via the API during the transition phase with validations of course > > - Keep this functionality configurable. > > > > We’re not trying to expand Polaris into a full IdP — just to provide a > > secure and practical bridge between versions. So the change seems fine to > > us. > > > > Happy to iterate on the proposal in a future sync > > > > On Wed, Jul 9, 2025 at 3:49 AM Dmitri Bourlatchkov <di...@apache.org> > > wrote: > > > > > Hi Arun, > > > > > > Thank you for starting this discussion! > > > > > > I did some poking about Keycloak and it looks like Keycloak allows > > > user-provided Client IDs. > > > > > > I think it should be fine for Polaris to accept user-provided Client > IDs > > in > > > the Principal management API. I suppose we may want to impose some > > > restrictions in terms of special characters, but in general a > > > previous Polaris Client ID should be valid as an input parameter when > > > creating a new Principal. > > > > > > I think it should also be fine for Polaris to accept user-provided > Client > > > Secrets (passwords) when creating Principals. > > > > > > That said, from my POV using the Admin Tool is still preferable for > > > migration use cases. > > > > > > My main argument in favour of the Admin Tool is that the whole > migration > > > process is a deployment type of activity when the Polaris service is > > > configured for the first time. Ideally, Polaris data would follow a > > backup > > > / restore process (not currently implemented) where the old instance's > > data > > > is exported into a file, which is then imported into the new instance > > > before it is started for the first time. > > > > > > I wonder how your tooling could obtain Principals' secrets from the old > > > Polaris instance for use as the new Principal creation request > parameter. > > > > > > Cheers, > > > Dmitri. > > > > > > On Tue, Jul 8, 2025 at 9:02 AM Arun Suri <arun.s...@fivetran.com > > .invalid> > > > wrote: > > > > > > > Hi all, > > > > > > > > Following up on the suggestion from the discussion here > > > > < > https://github.com/apache/polaris/issues/1929#issuecomment-3045487786 > > > > > > > — thank you for the feedback so far. > > > > > > > > We’re currently migrating our self-hosted Polaris service from > version > > > 0.9 > > > > (EclipseLink-based metastore) to version 1.0 (JDBC-based metastore). > As > > > > part of this transition, we need to preserve the existing `clientId` > > and > > > > `clientSecret` credentials for registered principals. > > > > These credentials are already embedded in customer workflows. > Rotating > > > them > > > > during migration would create disruptions and require cross-team > > > > coordination with clients — making both rollout and rollback > > > significantly > > > > more complex. > > > > > > > > We understand the security implications of allowing arbitrary > > credentials > > > > to be passed in an API request. That said, we believe this capability > > can > > > > be introduced safely and in a tightly controlled manner. For example: > > > > > > > > - Restricting this functionality to service admin only. > > > > - Ensuring all credential transmissions occur only over HTTPS > > > > - Clearly documenting that this is strictly for *migration/bootstrap > > use > > > > cases*, not for production use > > > > - Disabling this functionality by default in publicly hosted > > deployments > > > > - Ensuring credentials are never logged (e.g., in observability > systems > > > > like logs or traces) > > > > > > > > Our goal is not to weaken the system's security guarantees, but to > > > provide > > > > a practical and secure migration path where credential continuity is > > > > essential. Since there are a lot of DB schema changes involved, > Manual > > > > insertion into the metastore isn't ideal either, due to potential > > > > inconsistencies in hashing or salting logic across versions — > > increasing > > > > operational risk. > > > > > > > > *### Proposal* > > > > > > > > We propose extending the existing `createPrincipal` API to optionally > > > > accept `clientId` and `clientSecret` fields, with the above > safeguards > > in > > > > place. > > > > > > > > We would appreciate your feedback on this proposal and are happy to > > > > contribute a patch once there’s alignment. We’re also open to > > discussing > > > > this during the next Polaris Community Sync if helpful. > > > > > > > > Arun Suri > > > > > > > > Senior Software Engineer > > > > > > > > He/him/his > > > > > > > > Engineering | Fivetran > > > > arun.s...@fivetran.com > > > > fivetran.com <//fivetran.com> > > > > <http://www.fivetran.com> > > > > [image: facebook] <https://www.facebook.com/Fivetran/> [image: > > twitter] > > > > < > > > > > > > > > > https://twitter.com/fivetran?ref_src=twsrc%5Egoogle%7Ctwcamp%5Eserp%7Ctwgr%5Eauthor > > > > > > > > > [image: > > > > linkedin] <https://www.linkedin.com/company/fivetran> [image: > > instagram] > > > > <https://www.instagram.com/fivetran_ig/> > > > > > > > > > > > > > -- > > Arun Suri > > > > Senior Software Engineer > > > > He/him/his > > > > Engineering | Fivetran > > arun.s...@fivetran.com > > fivetran.com <//fivetran.com> > > <http://www.fivetran.com> > > [image: facebook] <https://www.facebook.com/Fivetran/> [image: twitter] > > < > > > https://twitter.com/fivetran?ref_src=twsrc%5Egoogle%7Ctwcamp%5Eserp%7Ctwgr%5Eauthor > > > > > [image: > > linkedin] <https://www.linkedin.com/company/fivetran> [image: instagram] > > <https://www.instagram.com/fivetran_ig/> > > >
