Hi Dmitri, + 1 for keeping the property bag at the API level and document well-known properties.
On the credential vs non-credential topic: my point is that in Iceberg's `LoadTableResult`, the `config` field could contain storage credentials as well, and that would be, afaict, perfectly valid: the credentials would then be valid for any prefix. Polaris, btw, does exactly that [1]. Therefore it would be more accurate, strictly speaking, to distinguish two scopes: "table defaults", and "prefix-specific overrides"; these would be orthogonal to the credential vs non-credential distinction, as illustrated in this example: [2]. Thanks, Alex [1]: https://github.com/apache/polaris/blob/8b108d6be7222a8ed78b1b2b70816ecbeea1b327/runtime/service/src/main/java/org/apache/polaris/service/catalog/iceberg/IcebergCatalogHandler.java#L876-L881 [2]: https://gist.github.com/adutra/1b6adb2c169c08bd91cd2a01ab4338d3 On Mon, Feb 23, 2026 at 4:12 PM Dmitri Bourlatchkov <[email protected]> wrote: > > Hi Alex, > > Re: config property mixing, my concern is not so much about prefix vs. > non-prefix config, but more about credentials vs. non-credentials > configuration. > > I believe distinguishing them at the API level will help implementations > treat them correctly too. Conversely, mixing credentials with other config > increases the chance of mis-handling them on the client side (e.g. logging > when they are not supposed to be logged). > > Your point about the fact that these properties will eventually be mixed on > the client side is quite valid. Therefore, I will not insist on separating > credential properties from ordinary config in the API. > > However, if the properties are mixed we should not use the name > "StorageAccessCredential" (note the last word) for their container, because > that (IMHO) would actually make the API spec confusing. I'd suggest naming > the container "StorageAccessConfiguration" (other names are welcome too). > > Re: well-structured properties objects - after thinking about it some more, > I believe your points about client compatibility prevail. I'd think clients > should be able to hangle new / unknown JSON properties, but it would > certainly be an overhead in many cases. > > I'm fine using a generic property bag as the type at the API level. > > I still believe we should document exact property names in the spec. For > that matter I propose moving property descriptions from Open API _comments_ > to Open API _description_ fields. This way, notes about property meaning > will not be lost after automated processing of the spec (e.g. by > SwaggerHub). WDYT? > > Thanks, > Dmitri. > > On Mon, Feb 23, 2026 at 7:27 AM Alexandre Dutra <[email protected]> wrote: > > > Hi all, > > > > > I do not think it is correct to mix the former with the latter. > > > > I don't think mixing general and prefix-specific configurations is > > inherently incorrect—it's a grey area. However, I agree that it is > > likely better practice to separate them. I believe the sensitivity > > level of the configuration should not be the deciding factor, as any > > configuration in a LoadTable response should be treated as sensitive. > > > > In Iceberg, the distinction in `LoadTableResult` between `config` and > > `storage-credentials.config` is primarily one of scope: the former > > applies broadly to the entire FileIO, and the latter is specific to an > > S3 client tied to a particular prefix. It's important to note, though, > > that any table-wide properties are ultimately merged with > > prefix-specific properties when the S3 client is created [1]. > > > > > Open API can be leveraged to report them as well-structured objects > > (JSON) > > > > The idea is interesting, especially since we control the specification > > and thus know the supported properties. > > > > However, this approach may complicate the evolution of the > > specification. Our guidance on evolution [2] states that new releases > > of Polaris should maintain compatibility with older clients. This > > requires that any addition or deprecation of properties must be done > > in a backward-compatible way. Removing or renaming properties would > > necessitate a major version bump for the specification. I am not > > convinced that the potential benefits outweigh these consequences. > > > > Thanks, > > Alex > > > > [1]: > > https://github.com/apache/iceberg/blob/fec9800bcc0c4073ca727f3b3bfdc2f34abb26a3/aws/src/main/java/org/apache/iceberg/aws/s3/S3FileIO.java#L411-L415 > > [2]: https://polaris.apache.org/releases/1.3.0/evolution/ > > > > > > On Sat, Feb 21, 2026 at 1:12 AM Dmitri Bourlatchkov <[email protected]> > > wrote: > > > > > > Hi All, > > > > > > I'm transferring some points from my GH comments [3826] here, for a wider > > > discussion. > > > > > > 1) Apparently some of the response properties relate to actual > > credentials > > > (key, expiry time), while others are more general configuration items > > (e.g. > > > the refresh endpoint). > > > > > > I do not think it is corrent to mix the former with the latter. Primarily > > > because of their different leak sensitivity levels but also because > > > `StorageAccessCredential` are provided as a list, and I wonder why we > > would > > > want to send multiple endpoint config entries (one for each location > > > prefix). > > > > > > 2) Currently properties are defined as an unstructured bag of key/value > > > pair. I think Open API can be leveraged to report them as well-structured > > > objects (JSON). > > > > > > Server code changes are required to add suport for new properties anyway. > > > It should not be too difficult to evolve the Open API types at the > > > same time. We've done it many times already (e.g. to support non-AWS S3 > > > storage). > > > > > > WDYT? > > > > > > Thanks, > > > Dmitri. > > > > > > [3826] https://github.com/apache/polaris/pull/3826/changes#r2829649503 > > > > > > On Sat, Feb 7, 2026 at 1:07 AM Jack Ye <[email protected]> wrote: > > > > > > > > How are clients supposed to know the meaning of credential > > properties in > > > > the "config" section? > > > > > > > > I think the contract is clear: since we are using the Iceberg REST > > model > > > > for StorageCredential, it must use the same configurations as the > > Iceberg > > > > credentials vending-related ones [1]. > > > > > > > > Based on the current Polaris implementation [2], the full list of > > configs > > > > is probably the following: > > > > > > > > S3: > > > > Credentials: > > > > - s3.access-key-id - AWS access key ID > > > > - s3.secret-access-key - AWS secret access key > > > > - s3.session-token - Temporary STS session token > > > > - s3.session-token-expires-at-ms - Token expiration timestamp (ms) > > > > Extra Properties: > > > > - s3.endpoint - S3 endpoint URI (optional) > > > > - s3.path-style-access - Path-style access flag (optional) > > > > - client.region - AWS region > > > > - aws.refresh-credentials-endpoint - Credential refresh endpoint > > > > (optional) > > > > > > > > GCS: > > > > Credentials: > > > > - gcs.oauth2.token - Downscoped OAuth2 access token > > > > - gcs.oauth2.token-expires-at - Token expiration timestamp (ms) > > > > Extra Properties: > > > > - gcs.oauth2.refresh-credentials-endpoint - Credential refresh > > endpoint > > > > (optional) > > > > > > > > Azure: > > > > Credentials: > > > > - adls.sas-token.<hostname> - SAS token keyed by storage account > > hostname > > > > - adls.sas-token-expires-at-ms.<hostname> - SAS token expiration (ms) > > > > Extra Properties: > > > > - adls.refresh-credentials-endpoint - Credential refresh endpoint > > > > (optional) > > > > > > > > It would be helpful to specify those at least in the generic tables > > spec, > > > > ideally also in the Iceberg REST spec, which currently only lists S3 > > > > configs and is already outdated. > > > > > > > > Best, > > > > Jack Ye > > > > > > > > [1] > > > > > > > > > > https://github.com/apache/polaris/blob/main/spec/iceberg-rest-catalog-open-api.yaml#L3299C1-L3306C1 > > > > [2] > > > > > > > > > > https://github.com/apache/polaris/blob/main/polaris-core/src/main/java/org/apache/polaris/core/storage/StorageAccessProperty.java > > > > > > > > > > > > On Fri, Feb 6, 2026 at 11:03 AM Eric Maynard <[email protected] > > > > > > > wrote: > > > > > > > > > > How are clients supposed to know the meaning of credential > > properties > > > > in > > > > > the "config" section? > > > > > > > > > > How are clients supposed to know the meaning of *any* properties > > written > > > > to > > > > > a generic table? > > > > > > > > > > That clients need to interpret the payload of a generic table > > response is > > > > > already intrinsic to the generic table design. True, adding > > credential > > > > > vending pushes generic table support towards the service being > > slightly > > > > > more opinionated about the generic table metadata (i.e. the location > > is > > > > now > > > > > implied to be a place that may require credentials to access) but as > > this > > > > > would be an opt-in for your generic tables I don't see this as a > > blocking > > > > > issue. > > > > > > > > > > --EM > > > > > > > > > > On Fri, Feb 6, 2026 at 10:57 AM Dmitri Bourlatchkov < > > [email protected]> > > > > > wrote: > > > > > > > > > > > Hi Yun, > > > > > > > > > > > > The proposal looks reasonable to me in general after a quick > > review of > > > > > the > > > > > > doc. > > > > > > > > > > > > I have one concern, though, which may be a whole can of worms, I'm > > > > afraid > > > > > > :) > > > > > > > > > > > > How are clients supposed to know the meaning of credential > > properties > > > > in > > > > > > the "config" section? The doc proposes to define it as a generic > > > > property > > > > > > bag. > > > > > > > > > > > > The example appears to use properties that Iceberg (java?) clients > > > > might > > > > > > use in a similar situation. However, the Generic Tables API is not > > > > > related > > > > > > to Iceberg in any way (AFAIK). > > > > > > > > > > > > Plus, I do not think these properties are well-defined even in > > Iceberg > > > > > > specifications. > > > > > > > > > > > > WDYT? > > > > > > > > > > > > Thanks, > > > > > > Dmitri. > > > > > > > > > > > > On Fri, Feb 6, 2026 at 1:29 PM yun zou <[email protected] > > > > > > > > wrote: > > > > > > > > > > > > > Hi All, > > > > > > > > > > > > > > Generic Tables have been available since Polaris 1.0.0 and have > > seen > > > > > > > growing interest from an increasing number of customers. > > > > > > > > > > > > > > However, the current Generic Table capability has some > > limitations. > > > > One > > > > > > > key gap is the lack of credential vending support. Without > > credential > > > > > > > vending, query engines must access tables using long-lived, > > static > > > > > cloud > > > > > > > credentials configured directly in the engine runtime, which > > limits > > > > > both > > > > > > > usability and security. > > > > > > > > > > > > > > To address this, we propose adding credential vending support for > > > > > Generic > > > > > > > Tables. This enhancement would allow a Polaris catalog to > > dynamically > > > > > > vend > > > > > > > short-lived, scoped storage credentials to query engines at > > runtime > > > > > when > > > > > > > accessing Generic Tables. > > > > > > > > > > > > > > The goals of this proposal are to: > > > > > > > > > > > > > > 1. > > > > > > > > > > > > > > Enable credential vending support for Generic Tables in > > Polaris > > > > > > > 2. > > > > > > > > > > > > > > Deliver an end-to-end experience for currently supported table > > > > > > > formats, including Delta, Hudi, and Lance > > > > > > > 3. > > > > > > > > > > > > > > Maintain consistency with the existing Iceberg credential > > vending > > > > > > model > > > > > > > > > > > > > > Please find the attached short design document with additional > > > > details. > > > > > > We > > > > > > > would appreciate your review and valuable feedback. > > > > > > > > > > > > > > link to google doc: > > > > > > > > > > > > > > > > > > > > > > > > https://docs.google.com/document/d/1QzTx4tcS23_mF-gc77GbTqtwuRHY5f_Aa_6E4VSKFU4/edit?tab=t.0#heading=h.rpqtaz73xt4v > > > > > > > > > > > > > > Best Regards, > > > > > > > > > > > > > > Yun > > > > > > > > > > > > > > > > > > > > > > > >
