Hi Yufei, I wonder if restricting setting the new GCS and Azure no-vending config properties similarly to how "stsUnavailable" is restricted [1] would alleviate your concerns?
WDYT? [1] https://github.com/apache/polaris/blob/b2b4a530e7c7fe104a1b37abb123bdc45bf178e3/runtime/service/src/main/java/org/apache/polaris/service/admin/PolarisServiceImpl.java#L162 Thanks, Dmitri. On Wed, Nov 4, 2026 at 2:48 PM Yufei Gu <[email protected]> wrote: > > I believe it is valuable for Polaris to provide a configuration option > for admin users to have better control in this situation. > > That makes sense. In that case, I think we could consider a catalog, > namespace, or table level config to control whether to vend > credentials. > > Given that we are moving toward a separate storage configuration > entity(pls find the dev mailing discussion link above), the same > storage configuration could be shared by multiple catalogs, > namespaces, or tables. Putting the vending flag in the storage > configuration would make it difficult to support different vending > behaviors for entities sharing the same configuration. > > I think whether to vend credentials is more of a behavior of the > catalog, namespace, or table itself, rather than a property of its > attached storage configuration. We shouldn't need to duplicate storage > configurations just to control credential vending differently. WDYT? > > Yufei > > > On Wed, Nov 4, 2026 at 11:25 AM Dmitri Bourlatchkov via dev > <[email protected]> wrote: > > > > Hi Yufei, > > > > Polaris will not vend credentials to clients if > X-Iceberg-Access-Delegation > > is not specified. However, it will vend if a client uses > > X-Iceberg-Access-Delegation. > > > > Polaris Servers cannot assume that all clients are well-behaving. I > believe > > it is valuable for Polaris to provide a configuration option for admin > > users to have better control in this situation. > > > > Secondary to that that, I think it should be possible to configure the > > Polaris Server itself to use ambient credentials for its own storage > access. > > > > This follows from your earlier proposal (to which I agree) to > > relegate SKIP_CREDENTIAL_SUBSCOPING_INDIRECTION to test-only use cases > [1]. > > > > [1] https://lists.apache.org/thread/1trcnbm04zzqztpxkzrzs1pvrlyd49bg > > > > Cheers, > > Dmitri. > > > > On Wed, Nov 4, 2026 at 1:53 PM Yufei Gu <[email protected]> wrote: > > > > > If engines have ambient credentials, they don't need to do anything. > > > For example, the default behavior of Spark[2] and Trino[1] does not > > > send the header `X-Iceberg-Access-Delegation`. Polaris won't vend any > > > credentials. This is a more flexible way for users, supporting mixed > > > use cases where some engines need credential vending and others don't. > > > > > > 1. Trino, explicitly documented as disabled by default: > > > - [REST catalog > > > documentation]( > > > > https://trino.io/docs/current/object-storage/metastores.html#rest-catalog > > > ): > > > `iceberg.rest-catalog.vended-credentials-enabled` “Defaults to > > > `false`.” > > > - [Header-setting > > > code]( > > > > https://github.com/trinodb/trino/blob/master/plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/catalog/rest/IcebergRestCatalogPropertiesProvider.java#L61-L67 > > > ): > > > adds `X-Iceberg-Access-Delegation: vended-credentials` only when > > > enabled; custom headers can also be configured. > > > > > > 2. Spark with Iceberg REST > > > - [SparkCatalog]( > > > > https://github.com/apache/iceberg/blob/main/spark/v4.0/spark/src/main/java/org/apache/iceberg/spark/SparkCatalog.java#L156-L162 > > > ): > > > forwards the configured catalog options to Iceberg. > > > - [RESTSessionCatalog]( > > > > https://github.com/apache/iceberg/blob/main/core/src/main/java/org/apache/iceberg/rest/RESTSessionCatalog.java#L181-L187 > > > ): > > > constructs HTTP headers using `RESTUtil.configHeaders(config)`. > > > - [RESTUtil.configHeaders]( > > > > https://github.com/apache/iceberg/blob/main/core/src/main/java/org/apache/iceberg/rest/RESTUtil.java#L433-L435 > > > ): > > > extracts explicitly configured `header.*` properties; it supplies no > > > default delegation value. > > > > > > Yufei > > > > > > On Wed, Nov 4, 2026 at 10:20 AM Dmitri Bourlatchkov via dev > > > <[email protected]> wrote: > > > > > > > > Hi All, > > > > > > > > From my POV the use case is clear and PR [5884] is ready to merge. > > > > > > > > If there are no further concerns, I propose merging it on Nov 6. > > > > > > > > Thanks, > > > > Dmitri. > > > > > > > > [5884] https://github.com/apache/polaris/pull/5884 > > > > > > > > On Mon, Nov 2, 2026 at 7:10 PM Dmitri Bourlatchkov <[email protected] > > > > > wrote: > > > > > > > > > Hi Yufei, > > > > > > > > > > > Could you share more details about the workload identity use > case? > > > > > > > > > > Clients (Engines) may have ambient credentials in the execution > env., > > > > > which allows them to access storage (e.g. GCS). > > > > > > > > > > In this case, Polaris does not have to vend credentials. However, > with > > > the > > > > > current GCS storage integration, Polaris cannot avoid vending > > > credentials. > > > > > > > > > > If the admin user elects to use workload identity for clients > (ambient > > > > > credentials), Polaris credential vending work is useless overhead. > > > Moreover > > > > > it may present a security gap by exposing different credentials to > > > clients, > > > > > which are supposed to authenticate via another (workload identity) > > > > > channel. I believe it is important for the admin user to be able to > > > > > minimize credential exposure via Polaris in this case. > > > > > > > > > > > I think we need a concrete use case to decide whether we need > this > > > [...] > > > > > > > > > > From my POV the use case I described above is self-sufficient. > > > > > > > > > > A similar practical use case relates to your reasonable and timely > > > > > proposal to limit the use of SKIP_CREDENTIAL_SUBSCOPING_INDIRECTION > > > [2]. > > > > > > > > > > If we are to take the next step and adapt our Integration Test to > act > > > as > > > > > examples of how to avoid relying on this flag, this results in > certain > > > > > coding difficulties, which are evident [1] in [5884]. > > > > > > > > > > > [...] where the setting should go. > > > > > > > > > > The natural place for a new setting that controls access to derived > > > > > credentials seems to be in the type-specific storage config > [5884], as > > > I > > > > > proposed in my original email in this thread. > > > > > > > > > > Do you have an alternative suggestion for how to address this use > case? > > > > > > > > > > Thanks, > > > > > Dmitri. > > > > > > > > > > [1] > https://github.com/apache/polaris/pull/5756#discussion_r4362688654 > > > > > > > > > > [2] > https://lists.apache.org/thread/1trcnbm04zzqztpxkzrzs1pvrlyd49bg > > > > > > > > > > [5756] https://github.com/apache/polaris/pull/5756 > > > > > > > > > > [5884] https://github.com/apache/polaris/pull/5884 > > > > > > > > > > On Mon, Nov 2, 2026 at 3:31 PM Yufei Gu <[email protected]> > wrote: > > > > > > > > > >> Hi Dmitri, > > > > >> > > > > >> Thanks for clarifying. My earlier comment was about clients > choosing > > > > >> not to request credential vending through > X-Iceberg-Access-Delegation. > > > > >> I understand that this is a server-side admin decision, which > differs > > > > >> from clients opting out. > > > > >> > > > > >> Could you share more details about the workload identity use > case? I’m > > > > >> still not clear why admins would need to disable credential > vending in > > > > >> that scenario. > > > > >> > > > > >> stsUnavailable is for S3-compatible storage without STS support. > If > > > > >> there’s a similar limitation for Azure/GCS, these settings make > sense > > > > >> to me. But if this is an admin policy, I’m not sure storage > > > > >> configuration is the right place, given that a storage config may > > > > >> apply to multiple tables or namespaces in the future (Rishi's > > > > >> proposal). > > > > >> > > > > >> I think we need a concrete use case to decide whether we need > this and > > > > >> where the setting should go. WDYT? > > > > >> > > > > >> > > > > >> Yufei > > > > >> > > > > >> On Fri, Oct 30, 2026 at 3:17 PM Dmitri Bourlatchkov via dev > > > > >> <[email protected]> wrote: > > > > >> > > > > > >> > Hi Yufei, > > > > >> > > > > > >> > > [...] we could use the header to achieve the same result. > > > > >> > > > > > >> > The proposed change allows admin users to configure Polaris to > avoid > > > > >> > vending credentials in Azure and GCS environments. It is a > > > deliberate > > > > >> > server-side admin decision. > > > > >> > > > > > >> > This was possible for S3 before. PR [5884] aligns Polaris > > > capabilities > > > > >> in > > > > >> > Azure and GCS with existing Polaris capabilities in S3 > environments. > > > > >> > > > > > >> > What alternative do you suggest? I'm not sure I can see that in > > > JB's or > > > > >> > Ayush's messages. > > > > >> > > > > > >> > Thanks, > > > > >> > Dmitri. > > > > >> > > > > > >> > [5884] https://github.com/apache/polaris/pull/5884 > > > > >> > > > > > >> > On Thu, Oct 29, 2026 at 7:30 PM Yufei Gu <[email protected]> > > > wrote: > > > > >> > > > > > >> > > Apologies for the late reply. > > > > >> > > > > > > >> > > Thanks Dmitri for raising this, and Vignesh for working onit. > As > > > > >> Ayush and > > > > >> > > JB mentioned, we could use the header to achieve the same > result. > > > Do > > > > >> we > > > > >> > > want to introduce another spec change, or am I missing some > use > > > cases? > > > > >> > > > > > > >> > > Yufei > > > > >> > > > > > > >> > > > > > > >> > > On Thu, Oct 29, 2026 at 2:05 PM Dmitri Bourlatchkov < > > > [email protected] > > > > >> > > > > > >> > > wrote: > > > > >> > > > > > > >> > > > Hi All, > > > > >> > > > > > > > >> > > > PR [5884] seems to be in a good shape. As far as I can > tell, all > > > > >> existing > > > > >> > > > points have been addressed. > > > > >> > > > > > > > >> > > > If there are no objections, I propose merging it on Oct 30. > > > > >> > > > > > > > >> > > > Thanks, > > > > >> > > > Dmitri. > > > > >> > > > > > > > >> > > > [5884] https://github.com/apache/polaris/pull/5884 > > > > >> > > > > > > > >> > > > On Tue, Oct 27, 2026 at 5:46 PM Dmitri Bourlatchkov < > > > > >> > > > [email protected]> wrote: > > > > >> > > > > > > > >> > > > > Hi JB, Ayush, > > > > >> > > > > > > > > >> > > > > Good points re: DefaultAccessDelegationModeResolver! > > > > >> > > > > > > > > >> > > > > Vignesh contributed PR [5884] to implement the new > properties. > > > > >> This PR > > > > >> > > > > adjusts the logic in DefaultAccessDelegationModeResolver > too. > > > > >> Kudos to > > > > >> > > > > Vugnesh! > > > > >> > > > > > > > > >> > > > > As far as I can tell, after [5884] if a client requests > > > > >> > > > > "vended-credentials" but the server is configured > > > > >> > > > > with skipCredentialDownscoping = true (for example), then > the > > > > >> client > > > > >> > > will > > > > >> > > > > receive a clear error message indicating that credential > > > vending > > > > >> is not > > > > >> > > > > available. > > > > >> > > > > > > > > >> > > > > This is related to the Iceberg API clarification vote > [1]. If > > > > >> that spec > > > > >> > > > > change passes in Iceberg, Polaris will be able to support > > > "none" > > > > >> as a > > > > >> > > > > vending mode for a better client config experience, IMHO. > > > > >> > > > > > > > > >> > > > > In any case, please review [5884]. I think it still makes > > > sense in > > > > >> > > > Polaris > > > > >> > > > > even without the Iceberg spec change. > > > > >> > > > > > > > > >> > > > > In Polaris code the main use case may be in tests, but > some > > > users > > > > >> may > > > > >> > > > want > > > > >> > > > > to consider the new options for workload identity-based > > > > >> deployments > > > > >> > > too. > > > > >> > > > > > > > > >> > > > > [1] > > > > >> https://lists.apache.org/thread/hc2764vozz7ygx5fjj64104hotrof0lk > > > > >> > > > > > > > > >> > > > > [5884] https://github.com/apache/polaris/pull/5884 > > > > >> > > > > > > > > >> > > > > Cheers, > > > > >> > > > > Dmitri. > > > > >> > > > > > > > > >> > > > > On Sun, Oct 25, 2026 at 5:53 AM Jean-Baptiste Onofré < > > > > >> [email protected]> > > > > >> > > > > wrote: > > > > >> > > > > > > > > >> > > > >> Hi all, > > > > >> > > > >> > > > > >> > > > >> +1 on the catalog-level > > > > >> AzureStorageConfigInfo.skipSasTokenGeneration > > > > >> > > > and > > > > >> > > > >> GcpStorageConfigInfo.skipCredentialDownscoping > properties. A > > > > >> typed, > > > > >> > > well > > > > >> > > > >> defined setting is a better home for "don't vend" than > the > > > > >> > > > >> SKIP_CREDENTIAL_SUBSCOPING_INDIRECTION flag, which only > > > applies > > > > >> at the > > > > >> > > > >> realm level. It also matches what stsUnavailable already > does > > > > >> for S3. > > > > >> > > > >> > > > > >> > > > >> I agree with Ayush on the access delegation mode. > > > > >> > > > >> > > > > >> > > > >> I think it would be great to clarify two points before > > > working > > > > >> on the > > > > >> > > > >> implementation PR: > > > > >> > > > >> - If a catalog opts out of vending and the client asks > only > > > for > > > > >> > > > >> vend-credentials, we should reject the request up front > with > > > a > > > > >> clear > > > > >> > > > >> error. > > > > >> > > > >> Failing later inside the vending path is harder to > diagnose > > > > >> imho. This > > > > >> > > > >> should apply to all three providers, so the S3 > stsUnavailable > > > > >> case > > > > >> > > > behaves > > > > >> > > > >> the same way. If changing the S3 behavior is out of the > scope > > > > >> for the > > > > >> > > > PR, > > > > >> > > > >> we can track as a follow-up, but we need consistency here > > > imho. > > > > >> > > > >> - Both properties should default to false, so existing > > > catalogs > > > > >> keep > > > > >> > > > >> vending credentials. They should be optional in the > > > Management > > > > >> API, so > > > > >> > > > >> this > > > > >> > > > >> is a backward compatible addition. We should also > document > > > them > > > > >> next > > > > >> > > to > > > > >> > > > >> stsUnavailable (in site/content/in-dev/unreleased/), and > say > > > > >> there > > > > >> > > that > > > > >> > > > >> they are meant for non-cloud or emulator deployments. > > > > >> > > > >> > > > > >> > > > >> Happy to review the PRs ;) > > > > >> > > > >> > > > > >> > > > >> Regards > > > > >> > > > >> JB > > > > >> > > > >> > > > > >> > > > >> On Fri, Oct 23, 2026 at 8:15 PM Dmitri Bourlatchkov < > > > > >> [email protected] > > > > >> > > > > > > > >> > > > >> wrote: > > > > >> > > > >> > > > > >> > > > >> > Hi All, > > > > >> > > > >> > > > > > >> > > > >> > As discovered during the review of PR 5756 [1], now > > > > >> > > > >> > that SKIP_CREDENTIAL_SUBSCOPING_INDIRECTION is not > > > recommended > > > > >> for > > > > >> > > > >> > mainstream use [3], deployments (e.g. tests) using > > > non-cloud > > > > >> options > > > > >> > > > for > > > > >> > > > >> > ADLS and GS (GCS) storage need a mechanism for opting > out > > > of > > > > >> minting > > > > >> > > > >> > session tokens and resolving to local (server and/or > > > client) > > > > >> > > > credentials > > > > >> > > > >> > for storage access. > > > > >> > > > >> > > > > > >> > > > >> > I propose adding the following properties at the > > > Management API > > > > >> > > level > > > > >> > > > >> [1]. > > > > >> > > > >> > > > > > >> > > > >> > * AzureStorageConfigInfo.skipSasTokenGeneration > > > > >> > > > >> > > > > > >> > > > >> > * GcpStorageConfigInfo.skipCredentialDownscoping > > > > >> > > > >> > > > > > >> > > > >> > Both names relate to the corresponding native APIs in > each > > > > >> > > technology. > > > > >> > > > >> > > > > > >> > > > >> > Setting these properties will effectively disable > > > credential > > > > >> > > vending. > > > > >> > > > >> This > > > > >> > > > >> > is similar to stsUnavailable in AwsStorageConfigInfo. > > > > >> > > > >> > > > > > >> > > > >> > The main purpose of the new properties is establishing > a > > > > >> > > catalog-level > > > > >> > > > >> opt > > > > >> > > > >> > in/out mechanism and providing well-defined user-level > > > > >> settings. > > > > >> > > > >> > > > > > >> > > > >> > PolarisStorageIntegration implementations still retain > > > value > > > > >> without > > > > >> > > > >> > credential vending because they potentially define > > > > >> non-credential > > > > >> > > > >> > properties in StorageAccessConfig. The latter case > > > currently > > > > >> applies > > > > >> > > > >> only > > > > >> > > > >> > to S3, but downstream customizations may utilize it > too. > > > > >> > > > >> > > > > > >> > > > >> > [1] > > > > >> > > > > > > https://github.com/apache/polaris/pull/5756#discussion_r4362688654 > > > > >> > > > >> > > > > > >> > > > >> > [2] > > > > >> > > > >> > > > > > >> > > > >> > > > > > >> > > > >> > > > > >> > > > > > > > >> > > > > > > >> > > > > https://github.com/apache/polaris/blob/5fafc67a832a8900fefb399abc00549d864470d6/spec/polaris-management-service.yml#L21 > > > > >> > > > >> > > > > > >> > > > >> > [3] > > > > >> > > > https://lists.apache.org/thread/1trcnbm04zzqztpxkzrzs1pvrlyd49bg > > > > >> > > > >> > > > > > >> > > > >> > Thoughts? > > > > >> > > > >> > > > > > >> > > > >> > Thanks, > > > > >> > > > >> > Dmitri. > > > > >> > > > >> > > > > > >> > > > >> > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > -- > > > > >> > > > > Dmitri Bourlatchkov > > > > >> > > > > Senior Staff Software Engineer, Dremio > > > > >> > > > > Dremio.com > > > > >> > > > > < > > > > >> > > > > > > > >> > > > > > > >> > > > > https://www.dremio.com/?utm_medium=email&utm_source=signature&utm_term=na&utm_content=email-signature&utm_campaign=email-signature > > > > >> > > > > > > > >> > > > / > > > > >> > > > > Follow Us on LinkedIn < > > > https://www.linkedin.com/company/dremio> > > > > >> / Get > > > > >> > > > > Started <https://www.dremio.com/get-started/> > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > The Agentic Lakehouse > > > > >> > > > > The only lakehouse built for agents, managed by agents > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > > > > >
