Hi Yufei, all, I've opened a PR with the API spec for storage configurations as a dedicated, catalog-scoped entity: https://github.com/apache/polaris/pull/5943. It is spec only and replaces #5721 and #5556.
In short: - Storage configurations are named entities under a catalog, each with its own settings and optional base location. They are immutable. The catalog's storageConfigInfo stays the default. - Storage mappings let an admin map the name of a namespace or table to a storage configuration, even before the object exists. Any create under that name then uses it, including plain Iceberg REST creates and CTAS, with no client changes. - A Polaris create API maps and creates in one call, using the Iceberg request and response bodies. - Each storage configuration can reuse server-side credentials by name through the existing storageName field from #3409. Credentials stay in server configuration. - Creating a storage configuration only runs static checks, because access is often granted afterwards. A separate validate call tests access to its locations on request. Yufei, mappings differ from the table property you suggested and from the pending bindings in my last email, so that's the part I'd most like feedback on. Which cloud identity a table gets is an admin's decision, and with a property anyone who can create a table would choose it. The PR spells out the rules for renames, stale mappings and cleanup. The storageName lookup exists only for AWS today. I'm also working on separate PRs to add it for GCS and Azure, and the spec assumes them. Please take a look. Thanks, Rishi On Mon, Oct 26, 2026 at 9:02 PM Yufei Gu <[email protected]> wrote: > Thanks, Rishi! This looks good overall. > > I'm good with Option B. Catalog scoped storage configurations also make > sense to me. I agree that having a dedicated entity gives us a cleaner > model for authorization, versioning, and future extensions. > > Regarding the binding issue, I wonder if we can keep things simpler by > allowing a storage configuration to be specified during table creation(for > example, specified in a table property, so that engines like Spark don't > have to adopt a new Polaris-specific plugin), while continuing to inherit > the parent's configuration by default. > > The pending binding approach seems more complicated, especially around > renames, stale bindings, and cleanup. I'd prefer to avoid introducing that > complexity unless we have a strong use case that requires it. > > I'm good with moving forward with the design doc based on Option B. > > > > Yufei > > > On Sun, Oct 25, 2026 at 8:49 PM Srinivas Rishindra <[email protected] > > > wrote: > > > Hi Yufei, > > > > Thanks for writing this up. I went through your points and I agree: let's > > skip Phase 1 and make storage configurations their own entities. > > > > On your points: > > > > 1. Default base location. This goes beyond create-table. With default > > settings a namespace can't have a custom location, so every namespace > > location is derived from the catalog's default-base-location. A > > namespace > > bound to an Azure configuration in an S3 catalog would get an s3:// > > location, and every table in it would fail the allowed-locations > check. > > The > > only workaround is turning on ALLOW_NAMESPACE_CUSTOM_LOCATION and > > setting > > each namespace's location by hand. An entity doesn't fix this by > itself > > either. I'd give each configuration an optional base location, use it > > for > > namespaces and tables bound before they're created (more below), and > > refuse > > to bind an existing namespace or table whose location isn't inside the > > configuration's allowed locations. > > > > 2, 3 and 5. Agreed. On 3: #5721 already moved to set/remove fields, but > > it's > > still one catalog entity with one version, so two admins editing > different > > configurations conflict. > > > > 4. Access control. Agreed, and this is the real cost of skipping Phase > > 1: we deferred AuthZ in June to unblock it, and now we need it up > front. > > I'd follow the policy model: create, read, write, drop and list > > privileges > > on the configuration, and attach as a pair (STORAGE_CONFIG_ATTACH on > the > > configuration plus a dedicated attach privilege on the namespace or > > table), > > all granted to catalog roles. It shouldn't be TABLE_WRITE_PROPERTIES, > > because TABLE_WRITE_DATA includes it and every writer could move a > > table to > > another identity. I'd leave out detach. Unlike a policy, unbinding a > > table > > that has data only points it at a configuration that can't reach its > > files. > > Changing a binding is a rebind, with the same location check as above. > > > > On your questions: > > > > 1. > > > > Skipping Phase 1: +1. > > 2. > > > > Option B. Policies already do most of what we need, but they fit > storage > > poorly. A policy lives under a namespace, and POLICY_CREATE is a > > namespace > > privilege. Its content is an untyped JSON string. Applicable policies > > are > > returned to anyone with TABLE_READ_PROPERTIES, so role ARNs and > external > > IDs would be visible to every table reader. And resolving them costs a > > lookup per path level on every loadTable. I'd copy the patterns > instead > > (attach checked on both sides, versioned updates) into a > STORAGE_CONFIG > > entity, and keep the binding as a configuration name on the namespace > or > > table, as you and Dmitri suggested in June. > > 3. > > > > Scope. Catalog-scoped. A configuration holds an identity (role ARN and > > external ID, Azure tenant, GCS service account) and a fence (allowed > > locations, base location). The identity is the part worth sharing. The > > fence has to stay per catalog, because the cross-catalog overlap check > > is > > what stops one catalog's admin from getting credentials for another > > catalog's files. A configuration shared across catalogs, locations > > included, would fail that check, and the only way around it is > > ALLOW_OVERLAPPING_CATALOG_URLS, which is realm-wide. So I'd start > > catalog-scoped with the identity inline. Later the identity can > become a > > realm-level object, managed by service_admin, that configurations > > reference > > by name. That's additive, and close to how Unity Catalog separates > > credentials from external locations. > > > > One more thing worth changing while we're at it: how a table ends up on a > > configuration other than its parent's. A binding needs an existing table, > > but > > Polaris writes the first metadata.json (and, for CTAS, vends the first > > credentials) with the parent's configuration. A Polaris-specific > > create-and-bind > > call would get around, but Spark, Trino and dbt can't use it directly. > > > > Instead (or in addition), an admin could bind a name ahead of time: > > "finance.ledger > > uses configuration finance". When a client then creates finance.ledger > > through the normal createTable, Polaris uses that configuration for the > > location, the first write and the credentials, and moves the binding onto > > the > > new table. Namespaces work the same way, which is what fixes point 1. The > > obvious concerns are stale bindings, renames, drop and re-create, and who > > owns a name that doesn't exist yet. I think each has an answer: a pending > > binding is used up at create and from then on behaves like any other > > binding, dropping the configuration deletes its pending bindings, and the > > namespace owns the names under it, so binding ahead needs the same > > privileges as binding the namespace. It does need a small table for > pending > > bindings. If that's too much, we could allow it for namespaces only; > tables > > then follow their namespace. > > > > If this sounds right, I'll update the design doc and share it here before > > opening a spec PR. I'll keep #5721 open until then. > > >
