Thanks a lot for driving this, Rulin! Left some comments in the doc. I think this is the right direction to make credential management more secure and more flexible.
Yufei On Thu, May 8, 2025 at 1:34 PM Rulin Xing <ru...@apache.org> wrote: > Hi folks, > > As Polaris expands its support for external services, such as federated > catalogs and cloud storage, it needs to securely access systems like AWS > S3, AWS Glue, Azure Storage, and others. This external access requires > Polaris to handle credentials correctly, whether they’re long-lived > credentials in self-managed deployments or temporary credentials in > multi-tenant SaaS setups. > > We've had several ongoing discussions about how credential handling should > evolve, especially in light of the work around SigV4 Auth for catalog > federation. > * [PR#1191] Fix updating the storage config > <https://github.com/apache/polaris/pull/1191> > * [PR#1506] Spec: Add SigV4 Auth Support for Catalog Federation > <https://github.com/apache/polaris/pull/1506> > * [Spec] Add SigV4 Auth Support for Catalog Federation > <https://lists.apache.org/thread/rlbxvw0xmzvlfm7pdh97bs3xvq7o8lmy> > > To frame the problem and proposed solutions, I’ve drafted a design doc: > Apache Polaris Creds Management Proposal > < > https://docs.google.com/document/d/1MAW87DtyHWPPNIEkUCRVUKBGjhh5bPn0GbtV7fifm30/edit?usp=sharing > > > > The proposal breaks the problem into four key areas: > 1. How Polaris gets vendor-specific service identity and credentials > (e.g., from server config and service context registry) > 2. How Polaris surfaces service identity info to users > (e.g., exposing userArn or consentUrl for trust policy setup) > 3. How Polaris injects service-managed identity fields into catalog or > storage configs > (e.g., using entity mutators at creation time) > 4. How Polaris retrieves temporary credentials to access external services > (e.g., via STS, with caching support) > > The goal is to unify credential handling across storage and connection > configs, support both SaaS and self-managed deployments, and cleanly > separate user-provided config from Polaris-managed properties. > > Would love to hear your thoughts, feedback, and suggestions. Happy to > refine based on feedback! > > Thanks, > Rulin >
