Hi, To follow up on Dmitri's point about credentials, there's already a PR <https://github.com/apache/polaris/pull/3409> up that is going to allow predefining named storage credentials in polaris config like the following:
- polaris.storage.aws.<storage-name>.access-key - polaris.storage.aws.<storage-name>.secret-key then storage configuration will simply refer to it by name and inherit credentials. I think that can go hand in hand with table-level overrides. Overriding each and every aws property for every table doesn't sound ideal. Defining a storage configuration upfront and referring to it by name should be a simpler solution. I can extend the scope of the PR above to allow predefining other aws properties as well like endpoint-url and region. Another point that came up in the discussion surrounding extra credentials is how to make sure anyone can't just hijack pre configured credentials. The simplest solution I see there is to ship off properties to OPA during catalog (and table) creation and allow users to write policies based on them. If we want to enable internal rbac to have a similar capability we can go further and move from config based storage definition to a separate `/storage-config` rest resource in management API that will come with necessary grants and permissions. On Thu, Feb 5, 2026 at 5:43 AM Dmitri Bourlatchkov <[email protected]> wrote: > Hi Srinivas, > > Thanks for the proposal. It looks good to me overall, a very timely feature > to add to Polaris. > > I added some comments in the doc and I see this topic on the Community Sync > agenda for Feb 5. Looking forward to discussing it online. > > I have three points to highlight: > > * Dealing with passwords probably connects to the Secrets Manager > discussion [1] > > * Persistence needs to consider non-RDBMS backends. OSS code has both > PostgreSQL and MongoDB, but private Persistence implementations are > possible too. I believe we need a proper SPI for this, not just a > relational schema example. > > * Associating entities (tables, namespaces) to Storage Configuration is > likely a plugin point that downstream projects may want to customize. I'd > propose making another SPI for this. This SPI is probably different from > the new Persistence SPI mentioned above since the concern here is not > persistence per se, but the logic of finding the right storage config. > > [1] https://lists.apache.org/thread/68r3gcx70f0qhbtz3w4zhb8f9s4vvw1f > > Cheers, > Dmitri. > > On Mon, Feb 2, 2026 at 4:18 PM Srinivas Rishindra <[email protected]> > wrote: > > > Hi all, > > > > We had an opportunity to discuss the community sprint last week. Based on > > that discussion, I have created a new design doc which I am attaching > here. > > In this design instead of passing credentials via table properties, this > > design introduces Inheritable Storage Configurations as a first-class > > feature. Please let me know your thoughts on the document. > > > > > > > https://docs.google.com/document/d/1hbDkE-w84Pn_112iW2vCnlDKPDtyg8flaYcFGjvD120/edit?usp=sharing > > > > > > On Mon, Jan 26, 2026 at 10:42 PM Yufei Gu <[email protected]> wrote: > > > > > Hi Srinivas, > > > > > > Thanks for sharing this proposal. Persisting long lived credentials > such > > as > > > an S3 secret access key directly in table properties raises significant > > > security concerns. Here is an alternative approach previously > discussed, > > > which enables storage configuration at the table or namespace level, > and > > it > > > is probably a more secure and promising direction overall. > > > > > > Yufei > > > > > > > > > On Mon, Jan 26, 2026 at 8:18 PM Srinivas Rishindra < > > [email protected] > > > > > > > wrote: > > > > > > > Dear All, > > > > > > > > I have developed a design proposal for Table-Level Storage Credential > > > > Overrides in Apache Polaris. > > > > > > > > The core objective is to allow specific storage properties to be > > defined > > > at > > > > the table level rather than the catalog level, enabling a single > > logical > > > > catalog to support tables across disparate storage systems. > Crucially, > > > the > > > > implementation ensures these overrides participate in the credential > > > > vending process to maintain secure, scoped access. > > > > > > > > I have also implemented a Proof of Concept (POC) pull request to > > > > demonstrate the idea. While the current MVP focuses on S3, I intend > to > > > > expand scope to include Azure and GCS pending community feedback. > > > > > > > > I look forward to your thoughts and suggestions on this proposal. > > > > > > > > Links: > > > > > > > > - Design Doc: Table-Level Storage Credential Overrides ( > > > > > > > > > > > > > > https://docs.google.com/document/d/1tf4N8GKeyAAYNoP0FQ1zT1Ba3P1nVGgdw3nmnhSm-u0/edit?usp=sharing > > > > ) > > > > - POC PR: https://github.com/apache/polaris/pull/3563 ( > > > > https://github.com/apache/polaris/pull/3563) > > > > > > > > Best regards, > > > > > > > > Srinivas Rishindra Pothireddi > > > > > > > > > >
