Hi Alex, I'm generally fine with not distinguishing credential properties at the API level. So property bags for table-defaults plus path-specific overrides sound reasonable to me.
However, Iceberg has a separate credentials section in LoadTableResult. Do you know the rationale for that? [1] https://github.com/apache/iceberg/blob/apache-iceberg-1.10.1/open-api/rest-catalog-open-api.yaml#L3321 Cheers, Dmitri. On Mon, Feb 23, 2026 at 12:07 PM Alexandre Dutra <[email protected]> wrote: > Hi Dmitri, > > + 1 for keeping the property bag at the API level and document > well-known properties. > > On the credential vs non-credential topic: my point is that in > Iceberg's `LoadTableResult`, the `config` field could contain storage > credentials as well, and that would be, afaict, perfectly valid: the > credentials would then be valid for any prefix. Polaris, btw, does > exactly that [1]. Therefore it would be more accurate, strictly > speaking, to distinguish two scopes: "table defaults", and > "prefix-specific overrides"; these would be orthogonal to the > credential vs non-credential distinction, as illustrated in this > example: [2]. > > Thanks, > Alex > > [1]: > https://github.com/apache/polaris/blob/8b108d6be7222a8ed78b1b2b70816ecbeea1b327/runtime/service/src/main/java/org/apache/polaris/service/catalog/iceberg/IcebergCatalogHandler.java#L876-L881 > [2]: https://gist.github.com/adutra/1b6adb2c169c08bd91cd2a01ab4338d3 > > > > > On Mon, Feb 23, 2026 at 4:12 PM Dmitri Bourlatchkov <[email protected]> > wrote: > > > > Hi Alex, > > > > Re: config property mixing, my concern is not so much about prefix vs. > > non-prefix config, but more about credentials vs. non-credentials > > configuration. > > > > I believe distinguishing them at the API level will help implementations > > treat them correctly too. Conversely, mixing credentials with other > config > > increases the chance of mis-handling them on the client side (e.g. > logging > > when they are not supposed to be logged). > > > > Your point about the fact that these properties will eventually be mixed > on > > the client side is quite valid. Therefore, I will not insist on > separating > > credential properties from ordinary config in the API. > > > > However, if the properties are mixed we should not use the name > > "StorageAccessCredential" (note the last word) for their container, > because > > that (IMHO) would actually make the API spec confusing. I'd suggest > naming > > the container "StorageAccessConfiguration" (other names are welcome too). > > > > Re: well-structured properties objects - after thinking about it some > more, > > I believe your points about client compatibility prevail. I'd think > clients > > should be able to hangle new / unknown JSON properties, but it would > > certainly be an overhead in many cases. > > > > I'm fine using a generic property bag as the type at the API level. > > > > I still believe we should document exact property names in the spec. For > > that matter I propose moving property descriptions from Open API > _comments_ > > to Open API _description_ fields. This way, notes about property meaning > > will not be lost after automated processing of the spec (e.g. by > > SwaggerHub). WDYT? > > > > Thanks, > > Dmitri. > > > > On Mon, Feb 23, 2026 at 7:27 AM Alexandre Dutra <[email protected]> > wrote: > > > > > Hi all, > > > > > > > I do not think it is correct to mix the former with the latter. > > > > > > I don't think mixing general and prefix-specific configurations is > > > inherently incorrect—it's a grey area. However, I agree that it is > > > likely better practice to separate them. I believe the sensitivity > > > level of the configuration should not be the deciding factor, as any > > > configuration in a LoadTable response should be treated as sensitive. > > > > > > In Iceberg, the distinction in `LoadTableResult` between `config` and > > > `storage-credentials.config` is primarily one of scope: the former > > > applies broadly to the entire FileIO, and the latter is specific to an > > > S3 client tied to a particular prefix. It's important to note, though, > > > that any table-wide properties are ultimately merged with > > > prefix-specific properties when the S3 client is created [1]. > > > > > > > Open API can be leveraged to report them as well-structured objects > > > (JSON) > > > > > > The idea is interesting, especially since we control the specification > > > and thus know the supported properties. > > > > > > However, this approach may complicate the evolution of the > > > specification. Our guidance on evolution [2] states that new releases > > > of Polaris should maintain compatibility with older clients. This > > > requires that any addition or deprecation of properties must be done > > > in a backward-compatible way. Removing or renaming properties would > > > necessitate a major version bump for the specification. I am not > > > convinced that the potential benefits outweigh these consequences. > > > > > > Thanks, > > > Alex > > > > > > [1]: > > > > https://github.com/apache/iceberg/blob/fec9800bcc0c4073ca727f3b3bfdc2f34abb26a3/aws/src/main/java/org/apache/iceberg/aws/s3/S3FileIO.java#L411-L415 > > > [2]: https://polaris.apache.org/releases/1.3.0/evolution/ > > > > > > > > > On Sat, Feb 21, 2026 at 1:12 AM Dmitri Bourlatchkov <[email protected]> > > > wrote: > > > > > > > > Hi All, > > > > > > > > I'm transferring some points from my GH comments [3826] here, for a > wider > > > > discussion. > > > > > > > > 1) Apparently some of the response properties relate to actual > > > credentials > > > > (key, expiry time), while others are more general configuration items > > > (e.g. > > > > the refresh endpoint). > > > > > > > > I do not think it is corrent to mix the former with the latter. > Primarily > > > > because of their different leak sensitivity levels but also because > > > > `StorageAccessCredential` are provided as a list, and I wonder why we > > > would > > > > want to send multiple endpoint config entries (one for each location > > > > prefix). > > > > > > > > 2) Currently properties are defined as an unstructured bag of > key/value > > > > pair. I think Open API can be leveraged to report them as > well-structured > > > > objects (JSON). > > > > > > > > Server code changes are required to add suport for new properties > anyway. > > > > It should not be too difficult to evolve the Open API types at the > > > > same time. We've done it many times already (e.g. to support non-AWS > S3 > > > > storage). > > > > > > > > WDYT? > > > > > > > > Thanks, > > > > Dmitri. > > > > > > > > [3826] > https://github.com/apache/polaris/pull/3826/changes#r2829649503 > > > > > > > > On Sat, Feb 7, 2026 at 1:07 AM Jack Ye <[email protected]> wrote: > > > > > > > > > > How are clients supposed to know the meaning of credential > > > properties in > > > > > the "config" section? > > > > > > > > > > I think the contract is clear: since we are using the Iceberg REST > > > model > > > > > for StorageCredential, it must use the same configurations as the > > > Iceberg > > > > > credentials vending-related ones [1]. > > > > > > > > > > Based on the current Polaris implementation [2], the full list of > > > configs > > > > > is probably the following: > > > > > > > > > > S3: > > > > > Credentials: > > > > > - s3.access-key-id - AWS access key ID > > > > > - s3.secret-access-key - AWS secret access key > > > > > - s3.session-token - Temporary STS session token > > > > > - s3.session-token-expires-at-ms - Token expiration timestamp > (ms) > > > > > Extra Properties: > > > > > - s3.endpoint - S3 endpoint URI (optional) > > > > > - s3.path-style-access - Path-style access flag (optional) > > > > > - client.region - AWS region > > > > > - aws.refresh-credentials-endpoint - Credential refresh endpoint > > > > > (optional) > > > > > > > > > > GCS: > > > > > Credentials: > > > > > - gcs.oauth2.token - Downscoped OAuth2 access token > > > > > - gcs.oauth2.token-expires-at - Token expiration timestamp (ms) > > > > > Extra Properties: > > > > > - gcs.oauth2.refresh-credentials-endpoint - Credential refresh > > > endpoint > > > > > (optional) > > > > > > > > > > Azure: > > > > > Credentials: > > > > > - adls.sas-token.<hostname> - SAS token keyed by storage account > > > hostname > > > > > - adls.sas-token-expires-at-ms.<hostname> - SAS token expiration > (ms) > > > > > Extra Properties: > > > > > - adls.refresh-credentials-endpoint - Credential refresh endpoint > > > > > (optional) > > > > > > > > > > It would be helpful to specify those at least in the generic tables > > > spec, > > > > > ideally also in the Iceberg REST spec, which currently only lists > S3 > > > > > configs and is already outdated. > > > > > > > > > > Best, > > > > > Jack Ye > > > > > > > > > > [1] > > > > > > > > > > > > > > https://github.com/apache/polaris/blob/main/spec/iceberg-rest-catalog-open-api.yaml#L3299C1-L3306C1 > > > > > [2] > > > > > > > > > > > > > > https://github.com/apache/polaris/blob/main/polaris-core/src/main/java/org/apache/polaris/core/storage/StorageAccessProperty.java > > > > > > > > > > > > > > > On Fri, Feb 6, 2026 at 11:03 AM Eric Maynard < > [email protected] > > > > > > > > > wrote: > > > > > > > > > > > > How are clients supposed to know the meaning of credential > > > properties > > > > > in > > > > > > the "config" section? > > > > > > > > > > > > How are clients supposed to know the meaning of *any* properties > > > written > > > > > to > > > > > > a generic table? > > > > > > > > > > > > That clients need to interpret the payload of a generic table > > > response is > > > > > > already intrinsic to the generic table design. True, adding > > > credential > > > > > > vending pushes generic table support towards the service being > > > slightly > > > > > > more opinionated about the generic table metadata (i.e. the > location > > > is > > > > > now > > > > > > implied to be a place that may require credentials to access) > but as > > > this > > > > > > would be an opt-in for your generic tables I don't see this as a > > > blocking > > > > > > issue. > > > > > > > > > > > > --EM > > > > > > > > > > > > On Fri, Feb 6, 2026 at 10:57 AM Dmitri Bourlatchkov < > > > [email protected]> > > > > > > wrote: > > > > > > > > > > > > > Hi Yun, > > > > > > > > > > > > > > The proposal looks reasonable to me in general after a quick > > > review of > > > > > > the > > > > > > > doc. > > > > > > > > > > > > > > I have one concern, though, which may be a whole can of worms, > I'm > > > > > afraid > > > > > > > :) > > > > > > > > > > > > > > How are clients supposed to know the meaning of credential > > > properties > > > > > in > > > > > > > the "config" section? The doc proposes to define it as a > generic > > > > > property > > > > > > > bag. > > > > > > > > > > > > > > The example appears to use properties that Iceberg (java?) > clients > > > > > might > > > > > > > use in a similar situation. However, the Generic Tables API is > not > > > > > > related > > > > > > > to Iceberg in any way (AFAIK). > > > > > > > > > > > > > > Plus, I do not think these properties are well-defined even in > > > Iceberg > > > > > > > specifications. > > > > > > > > > > > > > > WDYT? > > > > > > > > > > > > > > Thanks, > > > > > > > Dmitri. > > > > > > > > > > > > > > On Fri, Feb 6, 2026 at 1:29 PM yun zou < > [email protected] > > > > > > > > > > wrote: > > > > > > > > > > > > > > > Hi All, > > > > > > > > > > > > > > > > Generic Tables have been available since Polaris 1.0.0 and > have > > > seen > > > > > > > > growing interest from an increasing number of customers. > > > > > > > > > > > > > > > > However, the current Generic Table capability has some > > > limitations. > > > > > One > > > > > > > > key gap is the lack of credential vending support. Without > > > credential > > > > > > > > vending, query engines must access tables using long-lived, > > > static > > > > > > cloud > > > > > > > > credentials configured directly in the engine runtime, which > > > limits > > > > > > both > > > > > > > > usability and security. > > > > > > > > > > > > > > > > To address this, we propose adding credential vending > support for > > > > > > Generic > > > > > > > > Tables. This enhancement would allow a Polaris catalog to > > > dynamically > > > > > > > vend > > > > > > > > short-lived, scoped storage credentials to query engines at > > > runtime > > > > > > when > > > > > > > > accessing Generic Tables. > > > > > > > > > > > > > > > > The goals of this proposal are to: > > > > > > > > > > > > > > > > 1. > > > > > > > > > > > > > > > > Enable credential vending support for Generic Tables in > > > Polaris > > > > > > > > 2. > > > > > > > > > > > > > > > > Deliver an end-to-end experience for currently supported > table > > > > > > > > formats, including Delta, Hudi, and Lance > > > > > > > > 3. > > > > > > > > > > > > > > > > Maintain consistency with the existing Iceberg credential > > > vending > > > > > > > model > > > > > > > > > > > > > > > > Please find the attached short design document with > additional > > > > > details. > > > > > > > We > > > > > > > > would appreciate your review and valuable feedback. > > > > > > > > > > > > > > > > link to google doc: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://docs.google.com/document/d/1QzTx4tcS23_mF-gc77GbTqtwuRHY5f_Aa_6E4VSKFU4/edit?tab=t.0#heading=h.rpqtaz73xt4v > > > > > > > > > > > > > > > > Best Regards, > > > > > > > > > > > > > > > > Yun > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
