Hi all, Writing the following with my "nasty security guy" hat on:
Generally speaking, storing secrets is a quite sensitive topic that deserves a lot of attention upfront, during the initial implementation and for all changes related to it. What we already have in Polaris is IMHO strictly speaking not enough, because it lacks the ability to automatically rotate secrets and targeted logging/auditing functionality. It's overall okay-ish. With the ability to provide secrets comes another requirement to ensure that secrets are strong enough and new permission checks and feature flags, which make the code more complex. The mentioned requirements are the core domain of what IdPs do (for example the OSS projects Keycloak, Authelia). >From the Polaris's project PoV I think we should not go down that route but leverage IdPs to do this for us. Mistakes and oversights in the domain of secrets are common sources for vulnerabilities. On Mon, Jul 14, 2025 at 5:42 PM Dmitri Bourlatchkov <di...@apache.org> wrote: > > Hi Arun, > > After writing my previous email I realised that we might be able to support > your use case via REST API by implementing admin-base password reset [624]. > > The flow would be: > 1) create a principal using current ID (random client ID/secret) > 2) set your own client ID / secret via the password reset API (new) > > Would this work from your POV? > > As for me, it helps to isolate rare admin-level principal manipulation from > the mainstream principal API, which IMHO, is a good idea from a general > security perspective. > > Also, since we allow user-provided client IDs (after this change), it would > make sense to allow resetting them without re-creating the principal, hence > the new "reset" API. > > [624] https://github.com/apache/polaris/issues/624 > > Cheers, > Dmitri. > > On Mon, Jul 14, 2025 at 11:29 AM Dmitri Bourlatchkov <di...@apache.org> > wrote: > > > Thanks for volunteering to make a PR for this, Arun! Looking forward to it. > > > > As discussed, please consider securing the extra capability via (new) > > permission checks. I'd think it might be worth it to also have a feature > > flag to control the new functionality. > > > > Re: External IdP - most of the authentication code is already in `main`. > > There are a few remaining dangling pieces related to connecting external > > users to Polaris roles, though, IIRC. > > > > Cheers, > > Dmitri. > > > > On Mon, Jul 14, 2025 at 6:34 AM Arun Suri <arun.s...@fivetran.com.invalid> > > wrote: > > > >> Thanks for the detailed response Dimitri and Yufei! > >> > >> I agree with making the PR to support user-defined client ID and secret > >> via > >> the REST API, along with appropriate access checks and possibly > >> introducing > >> a new permission type/config. I will work on this > >> > >> REST fits better with our tooling as it has fewer dependencies and > >> complications compared to the JAR-based Admin CLI. We also believe > >> building > >> our migration logic in a neutral way (e.g., using the REST API) is more > >> robust—no matter how the tools evolve, the API remains the stable > >> contract. > >> > >> As for external IdP delegation, it's something we're open to exploring > >> down > >> the line, though we understand it's still relatively new in Polaris > >> > >> Thanks, > >> > >> On Sun, Jul 13, 2025 at 1:28 AM Yufei Gu <flyrain...@gmail.com> wrote: > >> > >> > The use case of passing secrets via REST is definitely valid, and this > >> use > >> > case is not the only one we should be considering. Other concrete > >> scenarios > >> > include: > >> > > >> > 1. Catalog federation, where Polaris needs to store credentials to > >> > connect to remote catalogs (e.g., Hive, Glue, Unity Catalog). > >> > 2. S3-compatible storage without STS support, where Polaris must > >> persist > >> > static access keys and secrets to enable read/write operations. > >> > > >> > Given these needs, I think it's the right time to formalize our > >> approach to > >> > secret management by integrating Polaris with established secret > >> managers > >> > such as HashiCorp Vault, AWS Secrets Manager, Azure Key Vault, and > >> Google > >> > Cloud Secret Manager. > >> > > >> > While using the admin tool to inject secrets is a workable short-term > >> > solution, it’s best treated as a stopgap. > >> > > >> > The good news is that the secret management interface was introduced in > >> the > >> > Polari core already, > >> > > >> > > >> https://github.com/apache/polaris/blob/main/polaris-core/src/main/java/org/apache/polaris/core/secrets/UserSecretsManager.java > >> > , > >> > we may just need to provide wrapper implementations for different secret > >> > managers. > >> > > >> > > >> > Yufei > >> > > >> > > >> > On Thu, Jul 10, 2025 at 12:52 PM Dmitri Bourlatchkov <di...@apache.org> > >> > wrote: > >> > > >> > > Thanks for providing more context, Arun! > >> > > > >> > > I do not object to adding user-provided client ID and secret to the > >> REST > >> > > API. However, I personally maintain my opinion that this kind of > >> > operation > >> > > fits better with the Admin Tool, given the current state of the > >> project. > >> > I > >> > > wonder what other community members think on this topic, too. > >> > > > >> > > If we go with updating the current REST API, then limiting access to > >> > > explicit client ID and secret parameters via access checks will > >> certainly > >> > > make sense. We may need a new permission type for this, I guess. > >> > > > >> > > Do you have the capacity to make a PR for this? > >> > > > >> > > Regarding the Admin Tool, is the difficulty in the fact that it is a > >> CLI > >> > > tool that requires a JVM and your existing tooling is based on > >> HTTP/REST > >> > > and is not written in java? Just trying to understand the overall use > >> > case > >> > > better. > >> > > > >> > > Your point about the external vault makes me wonder whether you might > >> be > >> > > interested in running an IdP server (e.g. keycloak) in your infra and > >> > > making Polaris delegate user management to that system. There's some > >> > > existing support for that, but I'm not sure if anyone tried it > >> end-to-end > >> > > without any custom code on the server side (it is certainly possible > >> with > >> > > custom code). > >> > > > >> > > Thanks, > >> > > Dmitri. > >> > > > >> > > On Thu, Jul 10, 2025 at 3:30 PM Arun Suri <arun.s...@fivetran.com > >> > .invalid> > >> > > wrote: > >> > > > >> > > > Hey Dmitri and Robert, > >> > > > > >> > > > > >> > > > *To clarify our use case further:* > >> > > > > >> > > > > >> > > > This isn't a one-time migration for us. We're migrating our > >> *customers* > >> > > > from > >> > > > Polaris 0.9 (EclipseLink) to Polaris 1.0 (JDBC) gradually. During > >> this > >> > > > process, we’ll be *running both catalog servers in parallel*, with > >> 1.0 > >> > > > acting as a *secondary/fallback* catalog. > >> > > > > >> > > > Our strategy involves: > >> > > > > >> > > > - Registering the same tables in both 0.9 and 1.0 > >> > > > - Using the *same *clientId* and *clientSecret in both catalogs > >> to > >> > > > ensure clients can authenticate seamlessly > >> > > > - Allowing us to *switch traffic between the two catalogs*, and > >> roll > >> > > > back instantly if needed > >> > > > > >> > > > This setup requires credential continuity — not just for migration, > >> but > >> > > to > >> > > > enable *safe rollback and zero-downtime cutover*. Using different > >> > > > credentials across catalogs versions would break this flow and > >> require > >> > > deep > >> > > > client coordination to rotate secrets, which is not feasible at > >> scale. > >> > > > > >> > > > > >> > > > *Regarding your question, Dmitri: *I wonder how your tooling could > >> > obtain > >> > > > Principals' secrets from the old > >> > > > > >> > > > Polaris instance for use as the new Principal creation request > >> > parameter > >> > > > > >> > > > - We store the credentials in an external Vault as well. So we are > >> not > >> > > > reading them from the old Polaris instance, but do have access to > >> them. > >> > > > > >> > > > > >> > > > We did consider raw table copying, but the differences in schema and > >> > > > hashing logic between 0.9 and 1.0 make that risky — and harder to > >> > > > validate/test it completely due to unknown risks. > >> > > > > >> > > > So our goal with this proposal is to: > >> > > > > >> > > > - Enable a *safe, service-admin only way* to inject known > >> > credentials > >> > > > via the API during the transition phase with validations of > >> course > >> > > > - Keep this functionality configurable. > >> > > > > >> > > > We’re not trying to expand Polaris into a full IdP — just to > >> provide a > >> > > > secure and practical bridge between versions. So the change seems > >> fine > >> > to > >> > > > us. > >> > > > > >> > > > Happy to iterate on the proposal in a future sync > >> > > > > >> > > > On Wed, Jul 9, 2025 at 3:49 AM Dmitri Bourlatchkov < > >> di...@apache.org> > >> > > > wrote: > >> > > > > >> > > > > Hi Arun, > >> > > > > > >> > > > > Thank you for starting this discussion! > >> > > > > > >> > > > > I did some poking about Keycloak and it looks like Keycloak allows > >> > > > > user-provided Client IDs. > >> > > > > > >> > > > > I think it should be fine for Polaris to accept user-provided > >> Client > >> > > IDs > >> > > > in > >> > > > > the Principal management API. I suppose we may want to impose some > >> > > > > restrictions in terms of special characters, but in general a > >> > > > > previous Polaris Client ID should be valid as an input parameter > >> when > >> > > > > creating a new Principal. > >> > > > > > >> > > > > I think it should also be fine for Polaris to accept user-provided > >> > > Client > >> > > > > Secrets (passwords) when creating Principals. > >> > > > > > >> > > > > That said, from my POV using the Admin Tool is still preferable > >> for > >> > > > > migration use cases. > >> > > > > > >> > > > > My main argument in favour of the Admin Tool is that the whole > >> > > migration > >> > > > > process is a deployment type of activity when the Polaris service > >> is > >> > > > > configured for the first time. Ideally, Polaris data would follow > >> a > >> > > > backup > >> > > > > / restore process (not currently implemented) where the old > >> > instance's > >> > > > data > >> > > > > is exported into a file, which is then imported into the new > >> instance > >> > > > > before it is started for the first time. > >> > > > > > >> > > > > I wonder how your tooling could obtain Principals' secrets from > >> the > >> > old > >> > > > > Polaris instance for use as the new Principal creation request > >> > > parameter. > >> > > > > > >> > > > > Cheers, > >> > > > > Dmitri. > >> > > > > > >> > > > > On Tue, Jul 8, 2025 at 9:02 AM Arun Suri <arun.s...@fivetran.com > >> > > > .invalid> > >> > > > > wrote: > >> > > > > > >> > > > > > Hi all, > >> > > > > > > >> > > > > > Following up on the suggestion from the discussion here > >> > > > > > < > >> > > https://github.com/apache/polaris/issues/1929#issuecomment-3045487786 > >> > > > > > >> > > > > > — thank you for the feedback so far. > >> > > > > > > >> > > > > > We’re currently migrating our self-hosted Polaris service from > >> > > version > >> > > > > 0.9 > >> > > > > > (EclipseLink-based metastore) to version 1.0 (JDBC-based > >> > metastore). > >> > > As > >> > > > > > part of this transition, we need to preserve the existing > >> > `clientId` > >> > > > and > >> > > > > > `clientSecret` credentials for registered principals. > >> > > > > > These credentials are already embedded in customer workflows. > >> > > Rotating > >> > > > > them > >> > > > > > during migration would create disruptions and require cross-team > >> > > > > > coordination with clients — making both rollout and rollback > >> > > > > significantly > >> > > > > > more complex. > >> > > > > > > >> > > > > > We understand the security implications of allowing arbitrary > >> > > > credentials > >> > > > > > to be passed in an API request. That said, we believe this > >> > capability > >> > > > can > >> > > > > > be introduced safely and in a tightly controlled manner. For > >> > example: > >> > > > > > > >> > > > > > - Restricting this functionality to service admin only. > >> > > > > > - Ensuring all credential transmissions occur only over HTTPS > >> > > > > > - Clearly documenting that this is strictly for > >> > *migration/bootstrap > >> > > > use > >> > > > > > cases*, not for production use > >> > > > > > - Disabling this functionality by default in publicly hosted > >> > > > deployments > >> > > > > > - Ensuring credentials are never logged (e.g., in observability > >> > > systems > >> > > > > > like logs or traces) > >> > > > > > > >> > > > > > Our goal is not to weaken the system's security guarantees, but > >> to > >> > > > > provide > >> > > > > > a practical and secure migration path where credential > >> continuity > >> > is > >> > > > > > essential. Since there are a lot of DB schema changes involved, > >> > > Manual > >> > > > > > insertion into the metastore isn't ideal either, due to > >> potential > >> > > > > > inconsistencies in hashing or salting logic across versions — > >> > > > increasing > >> > > > > > operational risk. > >> > > > > > > >> > > > > > *### Proposal* > >> > > > > > > >> > > > > > We propose extending the existing `createPrincipal` API to > >> > optionally > >> > > > > > accept `clientId` and `clientSecret` fields, with the above > >> > > safeguards > >> > > > in > >> > > > > > place. > >> > > > > > > >> > > > > > We would appreciate your feedback on this proposal and are > >> happy to > >> > > > > > contribute a patch once there’s alignment. We’re also open to > >> > > > discussing > >> > > > > > this during the next Polaris Community Sync if helpful. > >> > > > > > > >> > > > > > Arun Suri > >> > > > > > > >> > > > > > Senior Software Engineer > >> > > > > > > >> > > > > > He/him/his > >> > > > > > > >> > > > > > Engineering | Fivetran > >> > > > > > arun.s...@fivetran.com > >> > > > > > fivetran.com <//fivetran.com> > >> > > > > > <http://www.fivetran.com> > >> > > > > > [image: facebook] <https://www.facebook.com/Fivetran/> [image: > >> > > > twitter] > >> > > > > > < > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> https://twitter.com/fivetran?ref_src=twsrc%5Egoogle%7Ctwcamp%5Eserp%7Ctwgr%5Eauthor > >> > > > > > > > >> > > > > > [image: > >> > > > > > linkedin] <https://www.linkedin.com/company/fivetran> [image: > >> > > > instagram] > >> > > > > > <https://www.instagram.com/fivetran_ig/> > >> > > > > > > >> > > > > > >> > > > > >> > > > > >> > > > -- > >> > > > Arun Suri > >> > > > > >> > > > Senior Software Engineer > >> > > > > >> > > > He/him/his > >> > > > > >> > > > Engineering | Fivetran > >> > > > arun.s...@fivetran.com > >> > > > fivetran.com <//fivetran.com> > >> > > > <http://www.fivetran.com> > >> > > > [image: facebook] <https://www.facebook.com/Fivetran/> [image: > >> > twitter] > >> > > > < > >> > > > > >> > > > >> > > >> https://twitter.com/fivetran?ref_src=twsrc%5Egoogle%7Ctwcamp%5Eserp%7Ctwgr%5Eauthor > >> > > > > > >> > > > [image: > >> > > > linkedin] <https://www.linkedin.com/company/fivetran> [image: > >> > instagram] > >> > > > <https://www.instagram.com/fivetran_ig/> > >> > > > > >> > > > >> > > >> > >> > >> -- > >> Arun Suri > >> > >> Senior Software Engineer > >> > >> He/him/his > >> > >> Engineering | Fivetran > >> arun.s...@fivetran.com > >> fivetran.com <//fivetran.com> > >> <http://www.fivetran.com> > >> [image: facebook] <https://www.facebook.com/Fivetran/> [image: twitter] > >> < > >> https://twitter.com/fivetran?ref_src=twsrc%5Egoogle%7Ctwcamp%5Eserp%7Ctwgr%5Eauthor > >> > > >> [image: > >> linkedin] <https://www.linkedin.com/company/fivetran> [image: instagram] > >> <https://www.instagram.com/fivetran_ig/> > >> > >
