Hi Dmitri, Yufei, I marked #5972 ready for review.
I pushed a commit for Dmitri's GitHub comments and replied to the rest inline. The doc spacing nit turned out to be the docs generator, so I'll fix that in a separate small PR. Merging after 1.9.0 is fine by me. I'll add the changelog entry once the release is out. Thanks, Austen Get Outlook for Mac<https://aka.ms/GetOutlookForMac> From: Yufei Gu <[email protected]> Date: Thursday, November 5, 2026 at 18:09 To: [email protected] <[email protected]> Cc: Dmitri Bourlatchkov <[email protected]> Subject: Re: [DISCUSS] Cloudflare R2 support with scoped credential vending Hi Austen, the new config keys look good to me. Pls let us know once the PR is ready to review. Yufei On Tue, Nov 3, 2026 at 6:04 PM Dmitri Bourlatchkov via dev <[email protected]> wrote: > > Hi Austen, > > Thanks for pushing this feature forward! > > PR [5972] looks pretty nice (with minor comments in GH). I think it is good > for a general review. > > With the upcoming 1.9.0 release [1], it will probably be prudent to hold it > for a few days to avoid last minute feature changes and rushing the review. > I propose merging after the release. I hope it is ok. > > Cheers, > Dmitri. > > [1] https://lists.apache.org/thread/vvn28qykkfbvj07dhk5svnld532rjnty > > [5972] https://github.com/apache/polaris/pull/5972 > > > On Tue, Nov 3, 2026 at 4:41 PM Austen Tomek <[email protected]> > wrote: > > > Hi all, > > > > I opened #5972<https://github.com/apache/polaris/pull/5972> as a draft > > for the second part of the Cloudflare R2 work. It adds > > polaris-extensions-storage-cloudflare-r2, which implements the > > S3CredentialVendingMechanism SPI from #5513. The mechanism uses the > > R2_LOCAL_SIGNING name we agreed on in this thread. > > > > The server signs temporary R2 credentials locally with a parent token from > > its configuration. Credential vending makes no call to Cloudflare. The > > operator guide will follow in a separate PR. > > > > The extension is in the BOM and ships in the server bundle by default. It > > adds no new third-party dependencies to the bundle, so LICENSE and NOTICE > > stay unchanged. > > > > These keys become part of the public configuration: > > > > * polaris.storage.cloudflare-r2.access-key and > > polaris.storage.cloudflare-r2.secret-key set the default parent token. The > > environment variables are POLARIS_STORAGE_CLOUDFLARE_R2_ACCESS_KEY and > > POLARIS_STORAGE_CLOUDFLARE_R2_SECRET_KEY. > > * polaris.storage.cloudflare-r2.<storage>.access-key and .secret-key > > set named parent tokens. A catalog’s storageName selects one when > > RESOLVE_CREDENTIALS_BY_STORAGE_NAME is true. > > > > A realm opts in by adding R2_LOCAL_SIGNING to > > SUPPORTED_S3_CREDENTIAL_VENDING_MECHANISMS. The default stays [STS]. The PR > > description lists the credential limits operators need to know. > > > > I’ll leave the PR in draft for now to give this thread time to review the > > configuration keys. > > > > Austen > > > > > > Get Outlook for Mac<https://aka.ms/GetOutlookForMac> > > > > From: Dmitri Bourlatchkov <[email protected]> > > Date: Friday, October 2, 2026 at 17:25 > > To: [email protected] <[email protected]> > > Subject: Re: [DISCUSS] Cloudflare R2 support with scoped credential vending > > > > Hi Austen, > > > > R2_LOCAL_SIGNING works for me. > > > > Cheers, > > Dmitri. > > > > On Thu, Oct 1, 2026 at 5:30 PM Austen Tomek < > > [email protected]> > > wrote: > > > > > I like R2_LOCAL_SIGNING! It keeps R2 in the name which I think is > > > important as the token format varies by provider (R2’s is Cloudflare’s > > own > > > format). Another provider would likely need to provide its own value. > > > > > > Dmitri, does R2_LOCAL_SIGNING work for you too? > > > > > > > > > Get Outlook for Mac<https://aka.ms/GetOutlookForMac> > > > From: Yufei Gu <[email protected]> > > > Date: Thursday, October 1, 2026 at 15:59 > > > To: [email protected] <[email protected]> > > > Subject: Re: [DISCUSS] Cloudflare R2 support with scoped credential > > vending > > > > > > Hi Austen, > > > > > > Thanks! We should be good as long as the mechanisms are specified in the > > > spec so that both clients and servers won't be confused by a random > > string. > > > This approach ensures they are interoperable and easy to validate. > > > > > > Regarding the name, CLOUDFLARE_R2 sounds good. Here are a few > > alternatives > > > to express the mechanisms: > > > > > > - R2_LOCAL_SIGNING > > > - LOCAL_SIGNING > > > > > > LOCAL_SIGNING describes the method used for signing without reaching out > > to > > > the Security Token Service (STS), and it could potentially apply to other > > > storage providers beyond R2. > > > > > > What do you think? > > > > > > Yufei > > > > > > > > > On Thu, Oct 1, 2026 at 1:16 PM Dmitri Bourlatchkov <[email protected]> > > > wrote: > > > > > > > Hi Austen, > > > > > > > > Your plan matches my understanding. > > > > > > > > Also +1 to "CLOUDFLARE_R2" as the new mechanism name. > > > > > > > > Cheers, > > > > Dmitri. > > > > > > > > On Thu, Oct 1, 2026 at 3:56 PM Austen Tomek < > > > > [email protected]> > > > > wrote: > > > > > > > > > Thanks for the discussion today. My understanding of where we landed > > > was > > > > > to keep credentialVendingMechanism a string and have the spec define > > > the > > > > > values Apache Polaris accepts and what they mean. In this PR that is > > > just > > > > > STS and the empty value still signifies the server default. In PR 2 > > > I’ll > > > > be > > > > > adding CLOUDFLARE_R2 to the same list. The server already refuses any > > > > other > > > > > value with a 400, so it seems like this is just a documentation > > change. > > > > Let > > > > > me know if I am missing anything. > > > > > > > > > > Thanks, > > > > > Austen > > > > > > > > > > Get Outlook for Mac<https://aka.ms/GetOutlookForMac> > > > > > > > > > > From: Dmitri Bourlatchkov <[email protected]> > > > > > Date: Wednesday, September 30, 2026 at 12:31 > > > > > To: [email protected] <[email protected]> > > > > > Subject: Re: [DISCUSS] Cloudflare R2 support with scoped credential > > > > vending > > > > > > > > > > Hi Yufei, > > > > > > > > > > Thanks for providing the details! Replying inline. > > > > > > > > > > > When a client creates a catalog with credentialVendingMechanism: > > > "m1", > > > > > > what is the expected behavior? > > > > > > > > > > This depends on the meaning of "m1" and the specific server build. > > > > > > > > > > In Apache Polaris "m1" is not meaningful and is not supported. > > > > > The server will reject this value on the corresponding catalog > > > > > update request. This is handled in PolarisAdminService in the current > > > PR > > > > > state. > > > > > > > > > > > 1. The server supports it: This is the ideal scenario (happy > > path). > > > > > > > > > > Indeed. > > > > > > > > > > > 2. The server does not support it: This creates inconsistency if a > > > > > client > > > > > > that works with Server A fails on Server B. > > > > > > > > > > In my view this is not an inconsistency as different servers do not > > > have > > > > > to provide the same capabilities. > > > > > > > > > > Creating a catalog with a specific credentialVendingMechanism is an > > > > > administrative task. The admin user must be aware of what the server > > > > > supports. This is a deployment / configuration concern. Not a runtime > > > > > compatibility concern. > > > > > > > > > > > 3. The server supports "m1" differently: This is the worst-case > > > > > scenario, > > > > > > where the client's expectations conflict with the server's actual > > > > > behavior. > > > > > > > > > > This is a good point! > > > > > > > > > > My take is that if "m1" is defined in Apache Polaris, then all server > > > > > builds must implement it the same way. > > > > > > > > > > The definition will be expressed by means of > > > S3CredentialVendingMechanism > > > > > classes included in standard Apache Polaris releases (and hopefully > > > docs > > > > > too). > > > > > > > > > > Currently only "STS" and "DEFAULT" (falling back on STS) are defined > > > now. > > > > > > > > > > If "m1" is not defined in Apache Polaris, custom builds are free to > > > > assign > > > > > any meaning to it and do not have to be mutually compatible. This is > > > the > > > > > main point for extensibility. > > > > > > > > > > Note that custom builds are not "Apache Polaris" anymore and cannot > > > > > claim to be the "same" as Apache Polaris. This is a matter of ASF > > > > > trademarks. > > > > > Normally, custom builds say something like "Powered by Apache > > Polaris". > > > > > > > > > > Again, the admin user setting "m1" as a vending mechanism in a > > > particular > > > > > server must be aware of what that server provides and understand > > > > > the meaning of "m1" in that context. The onus to describe the meaning > > > of > > > > > "m1" to users in this case rests with the custom server developers. > > > > > > > > > > We can talk about preventing name overlaps, but I do not think > > > > establishing > > > > > any kind of name registry is worth the trouble. In this day and age, > > I > > > > > believe > > > > > all downstream project developers are aware of name overlap issues. > > > > > and will follow normal practices to avoid name collisions. > > > > > > > > > > I hope this clears your concerns. > > > > > > > > > > Cheers, > > > > > Dmitri. > > > > > > > > > > On Wed, Sep 30, 2026 at 1:02 PM Yufei Gu <[email protected]> > > wrote: > > > > > > > > > > > Hi Dmitri, > > > > > > > > > > > > When a client creates a catalog with credentialVendingMechanism: > > > "m1", > > > > > what > > > > > > is the expected behavior? > > > > > > > > > > > > Here is a suitaion if we allow a freeform string, the client might > > > > > > encounter three different scenarios depending on the server: > > > > > > 1. The server supports it: This is the ideal scenario (happy > > path). > > > > > > 2. The server does not support it: This creates inconsistency if a > > > > > client > > > > > > that works with Server A fails on Server B. > > > > > > 3. The server supports "m1" differently: This is the worst-case > > > > > scenario, > > > > > > where the client's expectations conflict with the server's actual > > > > > behavior. > > > > > > > > > > > > I don't think this is the situation we want users to run into. The > > > fix > > > > is > > > > > > quite simple, using a specified values instead of a freefrom > > string. > > > > > > > > > > > > Yufei > > > > > > > > > > > > > > > > > > On Tue, Sep 29, 2026 at 9:48 PM Jean-Baptiste Onofré < > > > [email protected]> > > > > > > wrote: > > > > > > > > > > > > > Hi all, > > > > > > > > > > > > > > Since concerns were raised, it's important to take a moment to > > > align > > > > > > > rather than rushing. That's the way we drive consensus :) > > > > > > > > > > > > > > On the technical question of string vs enum for > > > > > > > credentialVendingMechanism, here is my take: > > > > > > > > > > > > > > 1. This field is part of the Management API for catalog > > > > > > > administrators. Iceberg REST clients (Spark, Trino, ...) never > > see > > > or > > > > > > > interact with credentialVendingMechanism. They only consume > > > standard > > > > > > > vended credentials (s3.* session tokens/keys) during table > > > > > > > load/commit. There is no risk of breaking query engine clients > > > across > > > > > > > deployments. > > > > > > > 2. One of our primary goals when discussing S3-compatible stores > > > (R2, > > > > > > > MinIO, ...) was to keep credential vending pluggable via SPI > > rather > > > > > > > than hardcoding vendor-specific paths in core. If we use a closed > > > > enum > > > > > > > in the OpenAPI spec, it means: > > > > > > > * any new mechanism (including R2 in Phase 2 or MinIO) > > requires a > > > > > spec > > > > > > > change. > > > > > > > * any downstream deployment or plugin providing a custom > > > mechanism > > > > > > > would break OpenAPI-generated client SDKs due to strict enum > > > > > > > validation. > > > > > > > * runtime validation on the server (returning HTTP 400 with a > > > clear > > > > > > > message) combined with the realm allowlist provides the right > > > > > > > server-side guardrail without restricting extensibility. > > > > > > > 3. Yufei's concerns about a clear contract and avoiding confusion > > > is > > > > > > > valid. We can address contract clarity without closing the enum: > > > > > > > * we keep type: string, but document the well-known/standard > > > values > > > > > > > (STS, DEFAULT) explicitly in the description and examples in the > > > > > > > OpenAPI spec (as PR #5513 does). > > > > > > > * we ensure the server validates cleanly with > > case-insensitivity > > > > (or > > > > > > > canonical normalization) so administrators get immediate, > > > actionable > > > > > > > feedback on typos. > > > > > > > > > > > > > > PR #5513 does a solid job laying this foundation so Austen can > > move > > > > > > > forward with Phase 2. If we keep it as a string with > > > well-documented > > > > > > > standard values and server-siide validation, I think it's the > > right > > > > > > > balance between spec clarify and pluggability. > > > > > > > > > > > > > > Thoughts? > > > > > > > > > > > > > > Regards > > > > > > > JB > > > > > > > > > > > > > > On Wed, Sep 30, 2026 at 12:50 AM Dmitri Bourlatchkov < > > > > [email protected] > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > Hi Yufei, > > > > > > > > > > > > > > > > > > Conversely, an enum will require defining all possible > > > vending > > > > > > > > mechanisms in > > > > > > > > > > the Polaris API spec. > > > > > > > > > > > > > > > > > Yes. Otherwise, a client may failed in certain deployments. > > > > > > > > > > > > > > > > Could you describe this failure mode in terms of API > > > > request/response > > > > > > > flow? > > > > > > > > > > > > > > > > Thanks, > > > > > > > > Dmitri. > > > > > > > > > > > > > > > > On Tue, Sep 29, 2026 at 2:06 PM Yufei Gu <[email protected] > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > Conversely, an enum will require defining all possible > > > vending > > > > > > > > > mechanisms in > > > > > > > > > the Polaris API spec. > > > > > > > > > > > > > > > > > > Yes. Otherwise, a client may failed in certain deployments. > > > > > > > > > > > > > > > > > > > This will prevent downstream projects from using custom > > > vending > > > > > > > > > mechanisms. > > > > > > > > > > > > > > > > > > No. if downstream needs custom vending mechanisms, they can > > > > always > > > > > > > build > > > > > > > > > their own Polaris. > > > > > > > > > > > > > > > > > > Yufei > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Sep 29, 2026 at 11:01 AM Dmitri Bourlatchkov < > > > > > > [email protected] > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > Hi Yufei, > > > > > > > > > > > > > > > > > > > > > Using an enum will clarify the REST spec contract. > > > > > > > > > > > > > > > > > > > > I'm not sure I understand your point. Current PR clearly > > > > > describes > > > > > > > the > > > > > > > > > > meaning of "credentialVendingMechanism" and lists options > > > > > available > > > > > > > > > > out-of-the-box. > > > > > > > > > > > > > > > > > > > > What does an enum achieve, that (current) runtime > > validation > > > > does > > > > > > > not? > > > > > > > > > > > > > > > > > > > > Conversely, an enum will require defining all possible > > > vending > > > > > > > mechanisms > > > > > > > > > > in the Polaris API spec. This will prevent downstream > > > projects > > > > > from > > > > > > > using > > > > > > > > > > custom vending mechanisms. > > > > > > > > > > > > > > > > > > > > I believe Polaris as a project currently promotes > > downstream > > > > > > > > > > customizability. > > > > > > > > > > > > > > > > > > > > Cheers, > > > > > > > > > > Dmitri. > > > > > > > > > > > > > > > > > > > > On Tue, Sep 29, 2026 at 1:21 PM Yufei Gu < > > > [email protected] > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > I don't think we have consensus on the PR, esp. a client > > > > should > > > > > > NOT > > > > > > > > > > depend > > > > > > > > > > > on a specific deployment. Using an enum will clarify the > > > REST > > > > > > spec > > > > > > > > > > > contract. > > > > > > > > > > > > > > > > > > > > > > Yufei > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Sep 29, 2026 at 7:46 AM Dmitri Bourlatchkov < > > > > > > > [email protected]> > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > Hi All, > > > > > > > > > > > > > > > > > > > > > > > > PR [5513] has been in review for quite some time. I > > > believe > > > > > we > > > > > > > have > > > > > > > > > > > general > > > > > > > > > > > > consensus on the Management API changes and java code > > > > > changes. > > > > > > > > > > > > > > > > > > > > > > > > The only point that may still have divergent opinions > > is > > > > > > whether > > > > > > > to > > > > > > > > > use > > > > > > > > > > > > free text or enum for "credentialVendingMechanism". I > > > > > > personally > > > > > > > > > > support > > > > > > > > > > > > free text (rationale explained in previous comments). > > > This > > > > > > > > > conversation > > > > > > > > > > > has > > > > > > > > > > > > been dormant for a few days, so I assume lazy consensus > > > on > > > > > the > > > > > > > > > current > > > > > > > > > > > > state of the PR (free text). In any case switching from > > > > free > > > > > > > text to > > > > > > > > > > enum > > > > > > > > > > > > is a non-breaking API change before the next release in > > > > case > > > > > > the > > > > > > > > > > > consensus > > > > > > > > > > > > shifts and we need to adjust. We have about one month > > for > > > > > > > followup > > > > > > > > > API > > > > > > > > > > > > changes. > > > > > > > > > > > > > > > > > > > > > > > > The java code introduced in this PR is not rigid and > > can > > > > > change > > > > > > > at > > > > > > > > > any > > > > > > > > > > > time > > > > > > > > > > > > per our evolution guidelines [1]. > > > > > > > > > > > > > > > > > > > > > > > > I propose merging PR [5513] on Sept 30 unless fresh > > > > blocking > > > > > > > concerns > > > > > > > > > > are > > > > > > > > > > > > raised. > > > > > > > > > > > > > > > > > > > > > > > > [1] > > https://polaris.apache.org/releases/1.8.0/evolution/ > > > > > > > > > > > > > > > > > > > > > > > > [5513] https://github.com/apache/polaris/pull/5513 > > > > > > > > > > > > > > > > > > > > > > > > Cheers, > > > > > > > > > > > > Dmitri. > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Sep 29, 2026 at 9:30 AM Austen Tomek < > > > > > > > > > > > > [email protected]> wrote: > > > > > > > > > > > > > > > > > > > > > > > > > Hey all, > > > > > > > > > > > > > > > > > > > > > > > > > > Any chance I can get further reviews this week on the > > > PR? > > > > > I’d > > > > > > > like > > > > > > > > > to > > > > > > > > > > > get > > > > > > > > > > > > > moving onto Phase 2 here if there are no further > > issues > > > > to > > > > > > > address. > > > > > > > > > > > > > > > > > > > > > > > > > > Get Outlook for Mac<https://aka.ms/GetOutlookForMac> > > > > > > > > > > > > > > > > > > > > > > > > > > From: Dmitri Bourlatchkov <[email protected]> > > > > > > > > > > > > > Date: Thursday, September 24, 2026 at 17:33 > > > > > > > > > > > > > To: [email protected] <[email protected]> > > > > > > > > > > > > > Subject: Re: [DISCUSS] Cloudflare R2 support with > > > scoped > > > > > > > credential > > > > > > > > > > > > vending > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Yufei, > > > > > > > > > > > > > > > > > > > > > > > > > > > The REST spec is a contract between Polaris servers > > > and > > > > > > > clients > > > > > > > > > > [...] > > > > > > > > > > > > > > > > > > > > > > > > > > I tend to think that this statement is a bit too > > > strong. > > > > > The > > > > > > > > > > Management > > > > > > > > > > > > API > > > > > > > > > > > > > spec defines a protocol for clients and servers to > > > > > > communicate. > > > > > > > > > > > > > > > > > > > > > > > > > > > [...] This would help avoid a situation where a > > > Polaris > > > > > > > client > > > > > > > > > > works > > > > > > > > > > > > > > only with certain deployments. > > > > > > > > > > > > > > > > > > > > > > > > > > Since Polaris has been developed as a customizable > > > > > system, a > > > > > > > REST > > > > > > > > > > API > > > > > > > > > > > > > specification defined by Apache Polaris cannot > > control > > > > the > > > > > > > > > behaviour > > > > > > > > > > of > > > > > > > > > > > > all > > > > > > > > > > > > > possible downstream environments. The purpose of the > > > spec > > > > > is > > > > > > to > > > > > > > > > > ensure > > > > > > > > > > > > > protocol compatibility. > > > > > > > > > > > > > > > > > > > > > > > > > > Note that the value of `credentialVendingMechanism` > > has > > > > no > > > > > > > bearing > > > > > > > > > on > > > > > > > > > > > > > any logic clients need to execute. It is an opaque > > > > > parameter > > > > > > > > > > controlled > > > > > > > > > > > > > by the Polaris administrator. > > > > > > > > > > > > > > > > > > > > > > > > > > > The new credentialVendingMechanism field in the > > REST > > > > spec > > > > > > > looks > > > > > > > > > > > > > > reasonable. My only suggestion is to use an enum > > > > instead > > > > > > of a > > > > > > > > > > string. > > > > > > > > > > > > > [...] > > > > > > > > > > > > > > > > > > > > > > > > > > The above points are exactly the reason why the > > > > Management > > > > > > API > > > > > > > > > cannot > > > > > > > > > > > > > enumerate all possible vending mechanisms. > > > > > > > > > > > > > > > > > > > > > > > > > > The same applies to storage types too. However, we do > > > not > > > > > > have > > > > > > > any > > > > > > > > > > > > > practical > > > > > > > > > > > > > examples of 3rd party storage systems that go beyond > > > > > > > S3/GCS/ADLS > > > > > > > > > > (yet). > > > > > > > > > > > > > > > > > > > > > > > > > > Cheers, > > > > > > > > > > > > > Dmitri. > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Sep 24, 2026 at 4:38 PM Yufei Gu < > > > > > > [email protected] > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks Austen. It's fair that endpoints are not > > > > reliable > > > > > > > given > > > > > > > > > that > > > > > > > > > > > > some > > > > > > > > > > > > > > s3-compatible can use arbitrary hostnames. > > > > > > > > > > > > > > I also think stsUnavailable is fine to keep as-is. > > I > > > > > don’t > > > > > > > think > > > > > > > > > we > > > > > > > > > > > > need > > > > > > > > > > > > > to > > > > > > > > > > > > > > change its scope or repurpose it. > > > > > > > > > > > > > > > > > > > > > > > > > > > > The new credentialVendingMechanism field in the > > REST > > > > spec > > > > > > > looks > > > > > > > > > > > > > reasonable. > > > > > > > > > > > > > > My only suggestion is to use an enum instead of a > > > > string. > > > > > > The > > > > > > > > > REST > > > > > > > > > > > spec > > > > > > > > > > > > > is > > > > > > > > > > > > > > a contract between Polaris servers and clients, so > > > I’d > > > > > > > expect it > > > > > > > > > to > > > > > > > > > > > > > define > > > > > > > > > > > > > > the valid values and their meanings rather than > > > leaving > > > > > > them > > > > > > > up > > > > > > > > > to > > > > > > > > > > > each > > > > > > > > > > > > > > deployment. This would help avoid a situation > > where a > > > > > > Polaris > > > > > > > > > > client > > > > > > > > > > > > > works > > > > > > > > > > > > > > only with certain deployments. Deployment-specific > > > > > > mechanisms > > > > > > > > > > > shouldn’t > > > > > > > > > > > > > be > > > > > > > > > > > > > > part of the standard REST contract; users who need > > a > > > > > custom > > > > > > > > > > mechanism > > > > > > > > > > > > can > > > > > > > > > > > > > > always extend the server in their own deployment. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Either way the operator needs a per-realm switch > > > like > > > > > > > > > > > > > > SUPPORTED_S3_CREDENTIAL_VENDING_MECHANISMS to > > control > > > > > which > > > > > > > realm > > > > > > > > > > can > > > > > > > > > > > > > mint > > > > > > > > > > > > > > R2 credentials from the operators Cloudflare > > account. > > > > > > > > > > > > > > > > > > > > > > > > > > > > Could you clarify why we need a separate per-realm > > > > > switch? > > > > > > > Once > > > > > > > > > the > > > > > > > > > > > > > storage > > > > > > > > > > > > > > configuration is set at the catalog level to > > select a > > > > > > > specific > > > > > > > > > > > > mechanism, > > > > > > > > > > > > > > the catalog will have no choice but to support it. > > Do > > > > we > > > > > > need > > > > > > > > > > another > > > > > > > > > > > > > layer > > > > > > > > > > > > > > of gating? > > > > > > > > > > > > > > > > > > > > > > > > > > > > Yufei > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Sep 23, 2026 at 5:53 AM Austen Tomek < > > > > > > > > > > > > > > [email protected]> wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hey Yufei, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Apologies for missing your question. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I don’t think Endpoint is sufficient. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Endpoint matching only works where the hostname > > > > > > identifies > > > > > > > the > > > > > > > > > > > vendor > > > > > > > > > > > > > > like > > > > > > > > > > > > > > > R2’s public hostnames that are recognizable, > > MinIO > > > > and > > > > > > > other > > > > > > > > > > > > > self-hosted > > > > > > > > > > > > > > > S3-compatible stores use arbitrary hostnames so > > > there > > > > > > > would be > > > > > > > > > > > > nothing > > > > > > > > > > > > > to > > > > > > > > > > > > > > > match on. On top of that, if a match fails it > > would > > > > > just > > > > > > > > > silently > > > > > > > > > > > > fall > > > > > > > > > > > > > > back > > > > > > > > > > > > > > > to STS. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Right now the endpoint just says how you’ll > > connect > > > > not > > > > > > > how the > > > > > > > > > > > > > > > credentials are vended. There's endpoint and also > > > > > > > > > > endpointInternal > > > > > > > > > > > > for > > > > > > > > > > > > > > > server-side access, so it would be unclear which > > of > > > > the > > > > > > two > > > > > > > > > > > decides. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > For your other question, there is no second > > > mechanism > > > > > for > > > > > > > R2 > > > > > > > > > > today, > > > > > > > > > > > > but > > > > > > > > > > > > > > > for other self-hosted stores the same endpoint > > > could > > > > be > > > > > > > served > > > > > > > > > by > > > > > > > > > > > STS > > > > > > > > > > > > > or > > > > > > > > > > > > > > by > > > > > > > > > > > > > > > a different token mechanism chosen per catalog. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > stsUnavailable keeps it’s current behavior in the > > > PR > > > > > > (that > > > > > > > was > > > > > > > > > > > > > Prithvi’s > > > > > > > > > > > > > > > point earlier in the thread) > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Either way the operator needs a per-realm switch > > > like > > > > > > > > > > > > > > > SUPPORTED_S3_CREDENTIAL_VENDING_MECHANISMS to > > > control > > > > > > which > > > > > > > > > realm > > > > > > > > > > > can > > > > > > > > > > > > > > mint > > > > > > > > > > > > > > > R2 credentials from the operators Cloudflare > > > account. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > Austen > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Get Outlook for Mac< > > > https://aka.ms/GetOutlookForMac> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > From: Yufei Gu <[email protected]> > > > > > > > > > > > > > > > Date: Monday, September 21, 2026 at 17:53 > > > > > > > > > > > > > > > To: [email protected] < > > [email protected] > > > > > > > > > > > > > > > > > > > Subject: Re: [DISCUSS] Cloudflare R2 support with > > > > > scoped > > > > > > > > > > credential > > > > > > > > > > > > > > vending > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Austen, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks for the update. After looking at PR > > #5513, I > > > > > think > > > > > > > we > > > > > > > > > > should > > > > > > > > > > > > > first > > > > > > > > > > > > > > > clarify whether we need a new > > > > > credentialVendingMechanism > > > > > > > field > > > > > > > > > in > > > > > > > > > > > the > > > > > > > > > > > > > > > spec/config. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Austen, you mentioned that the R2 account ID and > > > > > > > jurisdiction > > > > > > > > > > > already > > > > > > > > > > > > > > come > > > > > > > > > > > > > > > from the endpoint. Could we also use the existing > > > > field > > > > > > > like > > > > > > > > > > > > > `endpoint`, > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > recognize R2 and select its credential-vending > > > > > mechanism, > > > > > > > while > > > > > > > > > > > > keeping > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > current stsUnavailable behavior unchanged? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Are there cases where the endpoint isn’t enough, > > or > > > > > where > > > > > > > users > > > > > > > > > > > need > > > > > > > > > > > > to > > > > > > > > > > > > > > > choose different vending mechanisms for the same > > > > > > > endpoint(e.g., > > > > > > > > > > R2) > > > > > > > > > > > > per > > > > > > > > > > > > > > > catalog? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Sep 16, 2026 at 7:24 AM Austen Tomek < > > > > > > > > > > > > > > > [email protected]> wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Yufei, Dimitri, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, both. PR 1 is ready for review: > > > > > > > > > > > > > > > > https://github.com/apache/polaris/pull/5513 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I pushed a revision last night that takes in > > > > > Dimitri’s > > > > > > > > > > comments. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > * > > > > > > > > > > > > > > > > I changed from issuer to > > > > credentialVendingMechanism. > > > > > > It’s > > > > > > > > > also > > > > > > > > > > > now > > > > > > > > > > > > a > > > > > > > > > > > > > > > plain > > > > > > > > > > > > > > > > string and the realm allowlist is > > > > > > > > > > > > > > > > SUPPORTED_S3_CREDENTIAL_VENDING_MECHANISMS, > > which > > > > > still > > > > > > > > > > defaults > > > > > > > > > > > to > > > > > > > > > > > > > > [STS] > > > > > > > > > > > > > > > > * > > > > > > > > > > > > > > > > Mechanisms are CDI beans found by their > > > identifier, > > > > > as > > > > > > > > > Dimitri > > > > > > > > > > > > > > proposed. > > > > > > > > > > > > > > > > * > > > > > > > > > > > > > > > > The CLOUDFLARE_R2 bean and the R2 vending code > > > will > > > > > > > follow in > > > > > > > > > > the > > > > > > > > > > > > > next > > > > > > > > > > > > > > PR > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > CI workflow is waiting approval to run. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > Austen > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Get Outlook for Mac< > > > > https://aka.ms/GetOutlookForMac> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > From: Dmitri Bourlatchkov <[email protected]> > > > > > > > > > > > > > > > > Date: Tuesday, September 15, 2026 at 20:00 > > > > > > > > > > > > > > > > To: [email protected] < > > > [email protected] > > > > > > > > > > > > > > > > > > > > > Subject: Re: [DISCUSS] Cloudflare R2 support > > with > > > > > > scoped > > > > > > > > > > > credential > > > > > > > > > > > > > > > vending > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi All, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Having getCloudflareR2Endpoint on > > > > > > > > > AwsStorageConfigurationInfo > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Yes, this does feel awkward. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > From my POV pursuing the option with the > > > > > > > "credentialIssuers" > > > > > > > > > > > > property > > > > > > > > > > > > > > > looks > > > > > > > > > > > > > > > > promising. This property is fairly generic and > > > fits > > > > > > > naturally > > > > > > > > > > > into > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > config, given that we already target multiple > > > > > backends. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Cheers, > > > > > > > > > > > > > > > > Dmitri. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Sep 15, 2026 at 8:48 PM Yufei Gu < > > > > > > > > > [email protected] > > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Austen, Dmitri, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Having getCloudflareR2Endpoint on > > > > > > > > > > AwsStorageConfigurationInfo > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > That does feel awkward. However, > > > > > > > > > AwsStorageConfigurationInfo > > > > > > > > > > > has > > > > > > > > > > > > > > > > supported > > > > > > > > > > > > > > > > > S3-compatible backends beyond AWS for a > > while, > > > so > > > > > > > there is > > > > > > > > > > > > > precedent > > > > > > > > > > > > > > > > (e.g., > > > > > > > > > > > > > > > > > MinIO, etc.). The underlying mismatch is that > > > we > > > > > call > > > > > > > the > > > > > > > > > > > storage > > > > > > > > > > > > > > type > > > > > > > > > > > > > > > > S3, > > > > > > > > > > > > > > > > > but its configuration mixes shared S3 > > settings > > > > with > > > > > > > > > > > AWS-specific > > > > > > > > > > > > > > ones: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > AwsStorageConfigurationInfo → storage type: > > S3 > > > > > > > > > > > > > > > > > ├── Common S3 settings > > > > > > > > > > > > > > > > > │ endpoint, region, pathStyleAccess, > > > > > > > allowedLocations > > > > > > > > > > > > > > > > > └── AWS-specific settings > > > > > > > > > > > > > > > > > roleArn, externalId, STS, KMS > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I’m open to separating the shared S3 settings > > > > from > > > > > > > > > > > > > provider-specific > > > > > > > > > > > > > > > > > configuration if that helps, but I don’t > > think > > > we > > > > > > need > > > > > > > to > > > > > > > > > > > settle > > > > > > > > > > > > > on a > > > > > > > > > > > > > > > > > broader refactor now. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Let me know when the PR is ready for review. > > > I’d > > > > be > > > > > > > happy > > > > > > > > > to > > > > > > > > > > > > take a > > > > > > > > > > > > > > > > closer > > > > > > > > > > > > > > > > > look. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Yufei > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Yufei > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mon, Sep 14, 2026 at 3:18 PM Dmitri > > > > > Bourlatchkov < > > > > > > > > > > > > > > [email protected]> > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Austen, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > +1 to adding "credentialIssues" or similar > > > > > > (renamed) > > > > > > > > > > property > > > > > > > > > > > > to > > > > > > > > > > > > > S3 > > > > > > > > > > > > > > > > > config. > > > > > > > > > > > > > > > > > > In fact, I was looking exactly for > > something > > > > like > > > > > > > that > > > > > > > > > last > > > > > > > > > > > > week > > > > > > > > > > > > > > but > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > idea did not quite condense in my mind :) > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I posted some preliminary comments on the > > > draft > > > > > PR. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Just for community awareness, I'd like to > > > > propose > > > > > > > here > > > > > > > > > that > > > > > > > > > > > we > > > > > > > > > > > > > use > > > > > > > > > > > > > > > CDI > > > > > > > > > > > > > > > > > for > > > > > > > > > > > > > > > > > > finding actual "issuer" / "mechanism" > > > > > > > implementations. > > > > > > > > > The > > > > > > > > > > > > > codebase > > > > > > > > > > > > > > > has > > > > > > > > > > > > > > > > > > many examples of this. The closest match in > > > > this > > > > > > > case is > > > > > > > > > > > > > > > > > > probably PolarisStorageIntegrationProvider. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > This way the API does not have to enumerate > > > all > > > > > > > possible > > > > > > > > > > > > values. > > > > > > > > > > > > > > > > > Downstream > > > > > > > > > > > > > > > > > > builds are free to add different > > > > implementations > > > > > > > without > > > > > > > > > > > having > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > change > > > > > > > > > > > > > > > > > > the API spec. The "allowed" list still > > makes > > > > > sense. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > In general, I like the direction this > > feature > > > > is > > > > > > > taking, > > > > > > > > > > > > > although I > > > > > > > > > > > > > > > > > expect > > > > > > > > > > > > > > > > > > PR reviews may take a few rounds :) > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Cheers, > > > > > > > > > > > > > > > > > > Dmitri. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mon, Sep 14, 2026 at 3:38 PM Austen > > Tomek > > > < > > > > > > > > > > > > > > > > > > [email protected]> wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi all, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Closing the loop since my last note and > > > > > answering > > > > > > > the > > > > > > > > > > > > questions > > > > > > > > > > > > > > > that > > > > > > > > > > > > > > > > > have > > > > > > > > > > > > > > > > > > > come up. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Dmitri's question on role ARNs: R2 > > doesn’t > > > > have > > > > > > > > > AWS-style > > > > > > > > > > > IAM > > > > > > > > > > > > > > roles > > > > > > > > > > > > > > > > or > > > > > > > > > > > > > > > > > > > ARNs. Our implementation signs temporary > > > > > > > credentials > > > > > > > > > > with a > > > > > > > > > > > > > > parent > > > > > > > > > > > > > > > > > > token’s > > > > > > > > > > > > > > > > > > > secret, scoped to a bucket and a set of > > > > > prefixes. > > > > > > > The > > > > > > > > > > > account > > > > > > > > > > > > > ID > > > > > > > > > > > > > > > and > > > > > > > > > > > > > > > > > > > jurisdiction come from the R2 endpoint. > > > > Putting > > > > > > the > > > > > > > > > > account > > > > > > > > > > > > ID > > > > > > > > > > > > > in > > > > > > > > > > > > > > > > > roleArn > > > > > > > > > > > > > > > > > > > or userArn would give those fields a > > > > different > > > > > > > meaning, > > > > > > > > > > so > > > > > > > > > > > > I’ve > > > > > > > > > > > > > > > left > > > > > > > > > > > > > > > > > them > > > > > > > > > > > > > > > > > > > absent for CLOUDFLARE_R2. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I pursued Option B. The S3 configuration > > > > gains > > > > > > one > > > > > > > new > > > > > > > > > > > field, > > > > > > > > > > > > > > > > > > > credentialIssuer, with values STS > > > > > > > > > > > > > > > > > > > (default) and CLOUDFLARE_R2 and a realm > > > > > allowlist > > > > > > > > > > > > > > > > > > > SUPPORTED_S3_CREDENTIAL_ISSUERS that > > > defaults > > > > > > > > > > > > > > > > > > > to [STS]. I left stsUnavailable untouched > > > > > > > (Prithvi) so > > > > > > > > > it > > > > > > > > > > > > still > > > > > > > > > > > > > > > means > > > > > > > > > > > > > > > > > > > "vend nothing". The > > > > > > > > > > > > > > > > > > > credentialIssuer is the explicit sign. > > The > > > R2 > > > > > > > config > > > > > > > > > > > rejects > > > > > > > > > > > > > > > > > > > stsUnavailable along with a host of other > > > STS > > > > > and > > > > > > > KMS > > > > > > > > > > > fields. > > > > > > > > > > > > > I'm > > > > > > > > > > > > > > > > > running > > > > > > > > > > > > > > > > > > > this in our non-production environment at > > > the > > > > > > > moment. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Draft PR for stage 1 (config field, API, > > > > > > allowlist, > > > > > > > > > > gates): > > > > > > > > > > > > > > > > > > > > > > https://github.com/apache/polaris/pull/5513. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I still see the appeal of a separate R2 > > > > > > > configuration, > > > > > > > > > as > > > > > > > > > > > > > Dmitri > > > > > > > > > > > > > > > > > > > suggested. My original reason for > > > separating > > > > it > > > > > > > was to > > > > > > > > > > keep > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > R2-specific > > > > > > > > > > > > > > > > > > > configuration and validation together. > > > Having > > > > > > > > > > > > > > > getCloudflareR2Endpoint > > > > > > > > > > > > > > > > > on > > > > > > > > > > > > > > > > > > > AwsStorageConfigurationInfo is one part > > I’d > > > > > still > > > > > > > like > > > > > > > > > > > > feedback > > > > > > > > > > > > > > on. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I wanted to put a working version of > > > Option B > > > > > in > > > > > > > front > > > > > > > > > of > > > > > > > > > > > > > > everyone > > > > > > > > > > > > > > > so > > > > > > > > > > > > > > > > > we > > > > > > > > > > > > > > > > > > > could discuss the actual changes. In my > > > > earlier > > > > > > > > > > > > implementation, > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > separate config also meant a separate > > > storage > > > > > > type > > > > > > > > > > sharing > > > > > > > > > > > > > > > S3FileIO. > > > > > > > > > > > > > > > > > That > > > > > > > > > > > > > > > > > > > was the tradeoff that led me to try > > Option > > > B. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Do the diffs change anyone’s lean here? > > And > > > > are > > > > > > the > > > > > > > > > names > > > > > > > > > > > > > > > > > > credentialIssuer > > > > > > > > > > > > > > > > > > > / STS / CLOUDFLARE_R2 / > > > > > > > SUPPORTED_S3_CREDENTIAL_ISSUERS > > > > > > > > > > > > > > acceptable? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > > > > Austen > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Get Outlook for Mac< > > > > > > > https://aka.ms/GetOutlookForMac> > > > > > > > > > > > > > > > > > > > From: Dmitri Bourlatchkov < > > > [email protected]> > > > > > > > > > > > > > > > > > > > Date: Sunday, September 13, 2026 at 19:50 > > > > > > > > > > > > > > > > > > > To: [email protected] < > > > > > > [email protected] > > > > > > > > > > > > > > > > > > > > > > > > > > > Subject: Re: [DISCUSS] Cloudflare R2 > > > support > > > > > with > > > > > > > > > scoped > > > > > > > > > > > > > > credential > > > > > > > > > > > > > > > > > > vending > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Building on top of Ayush's and JB's > > > > > suggestion, I > > > > > > > agree > > > > > > > > > > > that > > > > > > > > > > > > > > > > > jurisdiction > > > > > > > > > > > > > > > > > > > can live inside the existing endpoint > > > config > > > > > > field. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > That leaves only Account ID, so > > potentially > > > > we > > > > > > > could > > > > > > > > > > store > > > > > > > > > > > it > > > > > > > > > > > > > in > > > > > > > > > > > > > > > user > > > > > > > > > > > > > > > > > or > > > > > > > > > > > > > > > > > > > role ARN. Note that userArn is currently > > > > unused > > > > > > in > > > > > > > OSS > > > > > > > > > > > code. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > That should allow us to keep existing > > > > > > > > > > > > > AwsStorageConfigurationInfo > > > > > > > > > > > > > > > for > > > > > > > > > > > > > > > > > all > > > > > > > > > > > > > > > > > > > S3 storage backend, but will require > > > special > > > > > > > parsing of > > > > > > > > > > > > > user/role > > > > > > > > > > > > > > > ARN > > > > > > > > > > > > > > > > > > field > > > > > > > > > > > > > > > > > > > to distinguish R2 from the other systems. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I'm still leaning towards the separate > > > > > > > > > > > > > > R2StorageConfigurationInfo, > > > > > > > > > > > > > > > > but > > > > > > > > > > > > > > > > > > I'd > > > > > > > > > > > > > > > > > > > be ok with overloading user/role ARN too > > if > > > > > other > > > > > > > > > people > > > > > > > > > > > > prefer > > > > > > > > > > > > > > > > that. I > > > > > > > > > > > > > > > > > > > guess a key question in this context is > > > > whether > > > > > > R2 > > > > > > > has > > > > > > > > > > the > > > > > > > > > > > > > > concept > > > > > > > > > > > > > > > of > > > > > > > > > > > > > > > > > > role > > > > > > > > > > > > > > > > > > > ARN at all.... but TBH I have no idea > > about > > > > > > that. I > > > > > > > > > hope > > > > > > > > > > > > Austen > > > > > > > > > > > > > > > might > > > > > > > > > > > > > > > > > be > > > > > > > > > > > > > > > > > > > able to clarify that. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Cheers, > > > > > > > > > > > > > > > > > > > Dmitri. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Sun, Sep 13, 2026 at 9:18 AM Prithvi > > S < > > > > > > > > > > > > > > > > [email protected] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Austen, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Welcome, and thanks for starting with a > > > > > > > [DISCUSS]. > > > > > > > > > Best > > > > > > > > > > > way > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > make a > > > > > > > > > > > > > > > > > > > first > > > > > > > > > > > > > > > > > > > > contribution :) > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I think the thread is talking about two > > > > > layers, > > > > > > > and > > > > > > > > > we > > > > > > > > > > > can > > > > > > > > > > > > > keep > > > > > > > > > > > > > > > > both > > > > > > > > > > > > > > > > > > > > answers. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On the client side this should stay S3: > > > > s3:// > > > > > > > > > > locations, > > > > > > > > > > > > s3.* > > > > > > > > > > > > > > > > > > properties, > > > > > > > > > > > > > > > > > > > > S3FileIO. Your PyIceberg / DuckDB / > > > Iceberg > > > > > > Java > > > > > > > > > > results > > > > > > > > > > > > > > already > > > > > > > > > > > > > > > > show > > > > > > > > > > > > > > > > > > > that > > > > > > > > > > > > > > > > > > > > unmodified clients work, so a dedicated > > > > > > R2FileIO > > > > > > > does > > > > > > > > > > not > > > > > > > > > > > > > seem > > > > > > > > > > > > > > > > worth > > > > > > > > > > > > > > > > > > it. > > > > > > > > > > > > > > > > > > > > On the server side I lean toward > > Dmitri: > > > a > > > > > > > distinct > > > > > > > > > > > > > > > > > > > > R2StorageConfigurationInfo and storage > > > > > > > integration. > > > > > > > > > > > > Issuance > > > > > > > > > > > > > is > > > > > > > > > > > > > > > not > > > > > > > > > > > > > > > > > > STS, > > > > > > > > > > > > > > > > > > > > and folding Cloudflare fields into > > > > > > > > > > > > > AwsStorageConfigurationInfo > > > > > > > > > > > > > > > > mixes > > > > > > > > > > > > > > > > > > > > unrelated concepts. The middle ground > > to > > > > > avoid > > > > > > > is a > > > > > > > > > new > > > > > > > > > > > > type > > > > > > > > > > > > > > that > > > > > > > > > > > > > > > > is > > > > > > > > > > > > > > > > > > > still > > > > > > > > > > > > > > > > > > > > AWS-shaped underneath. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > One concrete caution if the Option B > > > > > prototype > > > > > > > stays > > > > > > > > > on > > > > > > > > > > > the > > > > > > > > > > > > > S3 > > > > > > > > > > > > > > > > > config: > > > > > > > > > > > > > > > > > > > > please do not overload stsUnavailable. > > > > Today > > > > > > that > > > > > > > > > flag > > > > > > > > > > > > skips > > > > > > > > > > > > > > > > > AssumeRole > > > > > > > > > > > > > > > > > > > and > > > > > > > > > > > > > > > > > > > > disables vending. R2 wants the > > opposite. > > > > > vend, > > > > > > > just > > > > > > > > > not > > > > > > > > > > > via > > > > > > > > > > > > > > STS. > > > > > > > > > > > > > > > so > > > > > > > > > > > > > > > > > > that > > > > > > > > > > > > > > > > > > > > flag would change meaning for existing > > > > MinIO > > > > > / > > > > > > > NetApp > > > > > > > > > > > > > catalogs. > > > > > > > > > > > > > > > If > > > > > > > > > > > > > > > > we > > > > > > > > > > > > > > > > > > > stay > > > > > > > > > > > > > > > > > > > > on S3 config, we need an explicit > > vending > > > > > > > signal; a > > > > > > > > > > > > dedicated > > > > > > > > > > > > > > R2 > > > > > > > > > > > > > > > > type > > > > > > > > > > > > > > > > > > > > avoids that trap. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On config surface, jurisdiction can > > live > > > in > > > > > > > endpoint > > > > > > > > > > > > > > (<account>. > > > > > > > > > > > > > > > > > > > > r2.cloudflarestorage.com vs > > > > > > <account>.eu.r2...). > > > > > > > > > > Account > > > > > > > > > > > > ID > > > > > > > > > > > > > is > > > > > > > > > > > > > > > > still > > > > > > > > > > > > > > > > > > > > needed > > > > > > > > > > > > > > > > > > > > for the JWT sub, so keeping an explicit > > > > > > > accountId and > > > > > > > > > > > > > deriving > > > > > > > > > > > > > > > > > audience > > > > > > > > > > > > > > > > > > > > from the endpoint seems enough. Parent > > > > tokens > > > > > > > should > > > > > > > > > > stay > > > > > > > > > > > > in > > > > > > > > > > > > > > > server > > > > > > > > > > > > > > > > > > > config > > > > > > > > > > > > > > > > > > > > via storageName. > > > > > > > > > > > > > > > > > > > > Local signing fits the vending model. > > > > > > > Fail-closed on > > > > > > > > > > > > > > cross-bucket > > > > > > > > > > > > > > > > and > > > > > > > > > > > > > > > > > > > mixed > > > > > > > > > > > > > > > > > > > > read/write grants is the right call. > > > Prefix > > > > > > > claims > > > > > > > > > > should > > > > > > > > > > > > > > follow > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > same > > > > > > > > > > > > > > > > > > > > path comparison Polaris already uses, > > so > > > we > > > > > do > > > > > > > not > > > > > > > > > > mint a > > > > > > > > > > > > > > > > credential > > > > > > > > > > > > > > > > > > > wider > > > > > > > > > > > > > > > > > > > > than the grant. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > +1 to small PRs: FileIO discriminator > > > > (#4486) > > > > > > > > > > separately > > > > > > > > > > > if > > > > > > > > > > > > > R2 > > > > > > > > > > > > > > > > needs > > > > > > > > > > > > > > > > > > it, > > > > > > > > > > > > > > > > > > > > then management API + Java config, then > > > > > > vending, > > > > > > > > > Python > > > > > > > > > > > CLI > > > > > > > > > > > > > on > > > > > > > > > > > > > > > its > > > > > > > > > > > > > > > > > own. > > > > > > > > > > > > > > > > > > > > Sketching MinIO / Backblaze against the > > > > same > > > > > > > fields > > > > > > > > > is > > > > > > > > > > a > > > > > > > > > > > > > useful > > > > > > > > > > > > > > > > > design > > > > > > > > > > > > > > > > > > > > check; a generic non-STS SPI can wait > > > for a > > > > > > > second > > > > > > > > > > > backend. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Regards, > > > > > > > > > > > > > > > > > > > > Prithvi S > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Sep 11, 2026 at 4:31 AM Dmitri > > > > > > > Bourlatchkov < > > > > > > > > > > > > > > > > > [email protected]> > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Austen, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks for starting this > > contribution! > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > After briefly looking through your PR > > > > [1] I > > > > > > > think > > > > > > > > > > > > > > > > > > > > > adding R2StorageConfigInfo makes > > sense. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Even though "aws" in > > > > > > > AwsStorageConfigurationInfo is > > > > > > > > > > > > longer > > > > > > > > > > > > > > > > implies > > > > > > > > > > > > > > > > > > AWS > > > > > > > > > > > > > > > > > > > > > services, and MinIO, Rust, Ozone are > > > > > > supported > > > > > > > just > > > > > > > > > > as > > > > > > > > > > > > > well, > > > > > > > > > > > > > > R2 > > > > > > > > > > > > > > > > > > appears > > > > > > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > > > > > be significantly different from > > > STS-based > > > > > S3 > > > > > > > > > systems > > > > > > > > > > > > that a > > > > > > > > > > > > > > > > > separate > > > > > > > > > > > > > > > > > > > > config > > > > > > > > > > > > > > > > > > > > > is probably the most natural way to > > > > support > > > > > > it > > > > > > > in > > > > > > > > > the > > > > > > > > > > > > > Polaris > > > > > > > > > > > > > > > > > > codebase. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > The alternative, I guess, is to add > > R2 > > > > > > > accountId > > > > > > > > > and > > > > > > > > > > > > > > > jurisdiction > > > > > > > > > > > > > > > > > as > > > > > > > > > > > > > > > > > > > > > optional properties to > > > > > > > AwsStorageConfigurationInfo > > > > > > > > > > (or > > > > > > > > > > > > > > overload > > > > > > > > > > > > > > > > > > > existing > > > > > > > > > > > > > > > > > > > > > properties), but that looks like > > mixing > > > > > > > unrelated > > > > > > > > > > > > concepts > > > > > > > > > > > > > > > > > together. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Adding a generic property bag to > > > > > > > > > > > > > AwsStorageConfigurationInfo > > > > > > > > > > > > > > is > > > > > > > > > > > > > > > > not > > > > > > > > > > > > > > > > > > > > > convenient because > > > > > > > > > > > PolarisStorageIntegrationProviderImpl > > > > > > > > > > > > > will > > > > > > > > > > > > > > > > > > probably > > > > > > > > > > > > > > > > > > > > have > > > > > > > > > > > > > > > > > > > > > to know how to interpret it in order > > to > > > > > > > redirect > > > > > > > > > > > > credential > > > > > > > > > > > > > > > > vending > > > > > > > > > > > > > > > > > > > > > requests to R2-specific code. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I wonder whether > > > > R2StorageConfigurationInfo > > > > > > and > > > > > > > > > > > > > > > > > > > > AwsStorageConfigurationInfo > > > > > > > > > > > > > > > > > > > > > (and corresponding "storage > > > integration" > > > > > > > classes) > > > > > > > > > > could > > > > > > > > > > > > > > share a > > > > > > > > > > > > > > > > > > common > > > > > > > > > > > > > > > > > > > > base > > > > > > > > > > > > > > > > > > > > > class for common properties (e.g. > > > > > endpoint). > > > > > > > Even > > > > > > > > > > > though > > > > > > > > > > > > > > > > "endpoint" > > > > > > > > > > > > > > > > > > > might > > > > > > > > > > > > > > > > > > > > > not be relevant to R2 in the cloud, > > the > > > > > fact > > > > > > > that > > > > > > > > > > > > S3FileIO > > > > > > > > > > > > > > uses > > > > > > > > > > > > > > > > it, > > > > > > > > > > > > > > > > > > and > > > > > > > > > > > > > > > > > > > > R2 > > > > > > > > > > > > > > > > > > > > > is accessed via S3FileIO means we > > > should > > > > > > > probably > > > > > > > > > > allow > > > > > > > > > > > > > users > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > > > > configure > > > > > > > > > > > > > > > > > > > > > the full set of FileIO properties > > just > > > in > > > > > > case. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I think it is fine for different > > > > > > > > > > > > PolarisStorageIntegration > > > > > > > > > > > > > > > > > > > > implementations > > > > > > > > > > > > > > > > > > > > > to produce the same kind of > > > > > > > StorageAccessConfig (S3 > > > > > > > > > > in > > > > > > > > > > > > this > > > > > > > > > > > > > > > case) > > > > > > > > > > > > > > > > > and > > > > > > > > > > > > > > > > > > > > thus > > > > > > > > > > > > > > > > > > > > > cause S3FileIO to be used by clients > > > > > > (including > > > > > > > > > > > Polaris' > > > > > > > > > > > > > own > > > > > > > > > > > > > > > > > storage > > > > > > > > > > > > > > > > > > > > access > > > > > > > > > > > > > > > > > > > > > paths). > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Changes in StorageTypeFileIO in [1] > > > look > > > > a > > > > > > bit > > > > > > > > > > > concerning > > > > > > > > > > > > > to > > > > > > > > > > > > > > > me. > > > > > > > > > > > > > > > > I > > > > > > > > > > > > > > > > > > > wonder > > > > > > > > > > > > > > > > > > > > > if this code could be refactored to > > > avoid > > > > > > > > > > dependencies > > > > > > > > > > > on > > > > > > > > > > > > > > > storage > > > > > > > > > > > > > > > > > > > config > > > > > > > > > > > > > > > > > > > > > types completely.... but TBH, I did > > not > > > > > look > > > > > > > too > > > > > > > > > > deeply > > > > > > > > > > > > > into > > > > > > > > > > > > > > > this > > > > > > > > > > > > > > > > > > > today. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Anticipating future PRs for this in > > the > > > > > > Polaris > > > > > > > > > repo, > > > > > > > > > > > I'd > > > > > > > > > > > > > > like > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > ask > > > > > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > > > > > separate Python code changes from > > java > > > > > > changes > > > > > > > > > since > > > > > > > > > > > > these > > > > > > > > > > > > > > > areas > > > > > > > > > > > > > > > > > > > usually > > > > > > > > > > > > > > > > > > > > > attract different reviewers :) > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > From my POV, management API and > > > matching > > > > > java > > > > > > > > > changes > > > > > > > > > > > can > > > > > > > > > > > > > be > > > > > > > > > > > > > > in > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > same > > > > > > > > > > > > > > > > > > > > > PR. I'd say docs should come later > > (to > > > > keep > > > > > > PRs > > > > > > > > > > small). > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > All in all, thank you again and I > > hope > > > > this > > > > > > > > > > > contribution > > > > > > > > > > > > > > lands > > > > > > > > > > > > > > > in > > > > > > > > > > > > > > > > > > > Polaris > > > > > > > > > > > > > > > > > > > > > :) > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > [1] > > > > > > > https://github.com/deepdishgary/polaris/pull/1 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Cheers, > > > > > > > > > > > > > > > > > > > > > Dmitri. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Sep 10, 2026 at 1:52 PM > > > > > Jean-Baptiste > > > > > > > > > Onofré > > > > > > > > > > < > > > > > > > > > > > > > > > > > > [email protected]> > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Austen, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Nice discussion! Thanks for that, > > and > > > > > > welcome > > > > > > > > > > aboard > > > > > > > > > > > :) > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I think it makes sense to have R2 > > > built > > > > > > > under the > > > > > > > > > > > > > existing > > > > > > > > > > > > > > S3 > > > > > > > > > > > > > > > > > > storage > > > > > > > > > > > > > > > > > > > > > > configuration. It matches how > > Polaris > > > > is > > > > > > > already > > > > > > > > > > > > > > structured: > > > > > > > > > > > > > > > > > > > > > > StorageType is keyed on the URI > > > scheme > > > > > > > (s3://, > > > > > > > > > > > s3a://), > > > > > > > > > > > > > and > > > > > > > > > > > > > > > > > > > > > > AwsStorageConfigurationInfo already > > > > > carries > > > > > > > > > > endpoint, > > > > > > > > > > > > > > > > > stsEndpoint, > > > > > > > > > > > > > > > > > > > > > > pathStyleAccess and, importantly, > > an > > > > > > > > > stsUnavailable > > > > > > > > > > > > flag. > > > > > > > > > > > > > > > > > > > > > > So, what you need is largely there > > > > > already: > > > > > > > > > correct > > > > > > > > > > > me > > > > > > > > > > > > if > > > > > > > > > > > > > > I'm > > > > > > > > > > > > > > > > > > wrong, > > > > > > > > > > > > > > > > > > > > > > R2 is essentially "S3 compatible > > > store > > > > > > where > > > > > > > STS > > > > > > > > > is > > > > > > > > > > > > > > > > unavailable, > > > > > > > > > > > > > > > > > > plus > > > > > > > > > > > > > > > > > > > > > > a way to actually vend credentials > > in > > > > > that > > > > > > > case". > > > > > > > > > > > Going > > > > > > > > > > > > > > this > > > > > > > > > > > > > > > > > route > > > > > > > > > > > > > > > > > > > > > > also keeps this work "isolated" > > from > > > > the > > > > > > > > > > > > > > > > FileIO-per-storage-type > > > > > > > > > > > > > > > > > > > > > > discussion. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > The interesting design question is > > a > > > > > > > pluggable > > > > > > > > > > > > credential > > > > > > > > > > > > > > > > vending > > > > > > > > > > > > > > > > > > > path > > > > > > > > > > > > > > > > > > > > > > for S3 compatible stores when STS > > is > > > > not > > > > > > > > > available, > > > > > > > > > > > > with > > > > > > > > > > > > > > > > > > Cloudflare's > > > > > > > > > > > > > > > > > > > > > > local signing as the first concrete > > > > > > > > > implementation, > > > > > > > > > > > > > rather > > > > > > > > > > > > > > > than > > > > > > > > > > > > > > > > > R2 > > > > > > > > > > > > > > > > > > > > > > specific code. It would be good to > > > > sketch > > > > > > how > > > > > > > > > MinIO > > > > > > > > > > > > would > > > > > > > > > > > > > > > slot > > > > > > > > > > > > > > > > > into > > > > > > > > > > > > > > > > > > > > > > the same config fields before the > > > > vending > > > > > > > path > > > > > > > > > > > hardens > > > > > > > > > > > > > > around > > > > > > > > > > > > > > > > R2. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I believe we have to check if > > > > Cloudflare > > > > > > > account > > > > > > > > > ID > > > > > > > > > > > and > > > > > > > > > > > > > > > > > > jurisdiction > > > > > > > > > > > > > > > > > > > > > > really need dedicated config > > fields, > > > or > > > > > can > > > > > > > we > > > > > > > > > use > > > > > > > > > > > the > > > > > > > > > > > > > > > endpoint > > > > > > > > > > > > > > > > > > value > > > > > > > > > > > > > > > > > > > > > > (with documentation)? Keeping the > > > > > > > configuration > > > > > > > > > > > surface > > > > > > > > > > > > > > > minimal > > > > > > > > > > > > > > > > > > will > > > > > > > > > > > > > > > > > > > > > > help the design generalize. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I would love to discuss two things: > > > > > > > > > > > > > > > > > > > > > > 1. Polaris shipping a Cloudflare > > API > > > > > token > > > > > > > and > > > > > > > > > > > locally > > > > > > > > > > > > > > > minting > > > > > > > > > > > > > > > > > > scoped > > > > > > > > > > > > > > > > > > > > > > credentials makes it effectively an > > > STS > > > > > for > > > > > > > R2, > > > > > > > > > > which > > > > > > > > > > > > > > > increases > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > > blast radius of a server > > compromise. > > > > Your > > > > > > > "reject > > > > > > > > > > > > rather > > > > > > > > > > > > > > than > > > > > > > > > > > > > > > > > > > boarden" > > > > > > > > > > > > > > > > > > > > > > handling of unrepresentable scopes > > is > > > > the > > > > > > > right > > > > > > > > > > call. > > > > > > > > > > > > > > > > > > > > > > 2. The PyIceberg/DuckDB/Java > > results > > > > are > > > > > > > > > promising. > > > > > > > > > > > > Just > > > > > > > > > > > > > > > > curious > > > > > > > > > > > > > > > > > > > about > > > > > > > > > > > > > > > > > > > > > > Spark or Trino. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > We love small, well-scoped PRs. I > > > would > > > > > > > suggest > > > > > > > > > to > > > > > > > > > > > > start > > > > > > > > > > > > > > with > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > S3 > > > > > > > > > > > > > > > > > > > > > > config and the management API as > > > first > > > > > > > shoot, and > > > > > > > > > > we > > > > > > > > > > > > can > > > > > > > > > > > > > > > follow > > > > > > > > > > > > > > > > > > with > > > > > > > > > > > > > > > > > > > > > > the credential vending > > > implementation. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Regards > > > > > > > > > > > > > > > > > > > > > > JB > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Sep 9, 2026 at 8:18 PM > > Austen > > > > > Tomek > > > > > > > > > > > > > > > > > > > > > > <[email protected]> > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi all, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I’m Austen Tomek from Chicago > > > Trading > > > > > > > Company. > > > > > > > > > We > > > > > > > > > > > use > > > > > > > > > > > > > > > Polaris > > > > > > > > > > > > > > > > > for > > > > > > > > > > > > > > > > > > > our > > > > > > > > > > > > > > > > > > > > > > Iceberg catalogs and we're > > evaluating > > > > > > > Cloudflare > > > > > > > > > R2 > > > > > > > > > > > as > > > > > > > > > > > > an > > > > > > > > > > > > > > > > object > > > > > > > > > > > > > > > > > > > store > > > > > > > > > > > > > > > > > > > > > (the > > > > > > > > > > > > > > > > > > > > > > no egress fees make it a very > > > > compelling > > > > > > > > > product). > > > > > > > > > > > I’d > > > > > > > > > > > > > like > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > > > discuss > > > > > > > > > > > > > > > > > > > > > > contributing R2 support and > > > credential > > > > > > > vending to > > > > > > > > > > > > Polaris > > > > > > > > > > > > > > > with > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > goal > > > > > > > > > > > > > > > > > > > > > of > > > > > > > > > > > > > > > > > > > > > > getting feedback before opening a > > > > > > > > > ready-for-review > > > > > > > > > > > > > upstream > > > > > > > > > > > > > > > PR. > > > > > > > > > > > > > > > > > > This > > > > > > > > > > > > > > > > > > > is > > > > > > > > > > > > > > > > > > > > > my > > > > > > > > > > > > > > > > > > > > > > first contribution, so trying to do > > > > this > > > > > > > right. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > R2 exposes S3-compatible APIs but > > > it > > > > > does > > > > > > > not > > > > > > > > > > > provide > > > > > > > > > > > > > AWS > > > > > > > > > > > > > > > > STS. > > > > > > > > > > > > > > > > > > > > > > Cloudflare instead supports these > > > > > > > short-lived, > > > > > > > > > > scoped > > > > > > > > > > > > > > > > credentials > > > > > > > > > > > > > > > > > > > > > generated > > > > > > > > > > > > > > > > > > > > > > by locally signed JWT with the > > server > > > > > > > holding the > > > > > > > > > > > > parent > > > > > > > > > > > > > > API > > > > > > > > > > > > > > > > > token > > > > > > > > > > > > > > > > > > > [1]. > > > > > > > > > > > > > > > > > > > > > > This allows us to retain Polaris’s > > > > > > > authorization > > > > > > > > > > and > > > > > > > > > > > > > > > > > > > credential-vending > > > > > > > > > > > > > > > > > > > > > > model while storing our table data > > in > > > > R2. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I have a prototype with the > > > following > > > > > > > design: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > * > > > > > > > > > > > > > > > > > > > > > > > An opt-in R2 storage type, with > > > > catalog > > > > > > > > > > > configuration > > > > > > > > > > > > > > > > > identifying > > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > > Cloudflare account and optional > > > > > > jurisdiction. > > > > > > > > > > Parent > > > > > > > > > > > > > > > > credentials > > > > > > > > > > > > > > > > > > > remain > > > > > > > > > > > > > > > > > > > > > in > > > > > > > > > > > > > > > > > > > > > > server configuration and can be > > > > selected > > > > > > > through > > > > > > > > > > > > > > storageName. > > > > > > > > > > > > > > > > > > > > > > > * > > > > > > > > > > > > > > > > > > > > > > > Local credential generation using > > > > > > > Cloudflare’s > > > > > > > > > > > > > documented > > > > > > > > > > > > > > > > > format. > > > > > > > > > > > > > > > > > > > > > > Credentials expire, are restricted > > to > > > > one > > > > > > > bucket, > > > > > > > > > > and > > > > > > > > > > > > > carry > > > > > > > > > > > > > > > > > prefix > > > > > > > > > > > > > > > > > > > and > > > > > > > > > > > > > > > > > > > > > > access scopes derived from > > Polaris’s > > > > > > location > > > > > > > > > > grants. > > > > > > > > > > > > The > > > > > > > > > > > > > > > > > > integration > > > > > > > > > > > > > > > > > > > > > uses > > > > > > > > > > > > > > > > > > > > > > Polaris’s existing credential > > cache. > > > > > > > > > > > > > > > > > > > > > > > * > > > > > > > > > > > > > > > > > > > > > > > Credentials returned through the > > > > > existing > > > > > > > > > Iceberg > > > > > > > > > > > > REST > > > > > > > > > > > > > > > > vending > > > > > > > > > > > > > > > > > > flow > > > > > > > > > > > > > > > > > > > > as > > > > > > > > > > > > > > > > > > > > > > s3.* properties, including the > > > > endpoint, > > > > > > > session > > > > > > > > > > > token, > > > > > > > > > > > > > and > > > > > > > > > > > > > > > > > expiry. > > > > > > > > > > > > > > > > > > > > > > Compatible clients use their > > existing > > > > S3 > > > > > > > FileIO. > > > > > > > > > > > > > > > > > > > > > > > * > > > > > > > > > > > > > > > > > > > > > > > Conservative handling of scopes > > the > > > > > > > > > > implementation > > > > > > > > > > > > > cannot > > > > > > > > > > > > > > > > > > > represent: > > > > > > > > > > > > > > > > > > > > > > cross-bucket and mixed read/write > > > > grants > > > > > > are > > > > > > > > > > rejected > > > > > > > > > > > > > > rather > > > > > > > > > > > > > > > > than > > > > > > > > > > > > > > > > > > > > > > broadened. R2 is excluded from the > > > > > default > > > > > > > > > > > > > > > > > supported-storage-types > > > > > > > > > > > > > > > > > > > > list. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > The main design questions I feel > > > this > > > > > > runs > > > > > > > > > into: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > * > > > > > > > > > > > > > > > > > > > > > > > Should this be a distinct storage > > > > type > > > > > > or a > > > > > > > > > > > > > > > > credential-vending > > > > > > > > > > > > > > > > > > > > > mechanism > > > > > > > > > > > > > > > > > > > > > > selected through the existing S3 > > > > > > > configuration? > > > > > > > > > > > > > > > > > > > > > > > * > > > > > > > > > > > > > > > > > > > > > > > I chose a separate type because > > the > > > > > > > identity, > > > > > > > > > > > > > credential > > > > > > > > > > > > > > > > > > > generation, > > > > > > > > > > > > > > > > > > > > > and > > > > > > > > > > > > > > > > > > > > > > scoping model differ from AWS STS, > > as > > > > > this > > > > > > > seemed > > > > > > > > > > > more > > > > > > > > > > > > > > > > > appropriate > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > There is a draft preview PR > > against > > > > my > > > > > > fork > > > > > > > > > [2], > > > > > > > > > > > > > > including > > > > > > > > > > > > > > > > > > > > > > implementation, management API > > > changes, > > > > > CLI > > > > > > > > > > support, > > > > > > > > > > > > > tests, > > > > > > > > > > > > > > > and > > > > > > > > > > > > > > > > > > > > > > documentation. It also discloses > > the > > > AI > > > > > > > > > assistance > > > > > > > > > > > used > > > > > > > > > > > > > > > during > > > > > > > > > > > > > > > > > > > > > > implementation. I leaned heavily on > > > > Fable > > > > > > > 5.1 for > > > > > > > > > > > this > > > > > > > > > > > > > > > > > > implementation > > > > > > > > > > > > > > > > > > > > so > > > > > > > > > > > > > > > > > > > > > I > > > > > > > > > > > > > > > > > > > > > > want to be upfront about it. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > We are running the implementation > > > in > > > > a > > > > > > > > > > > non-production > > > > > > > > > > > > > > > > > environment > > > > > > > > > > > > > > > > > > > > right > > > > > > > > > > > > > > > > > > > > > > now as a POC. Validation includes > > > live > > > > R2 > > > > > > > > > > > reads/writes, > > > > > > > > > > > > > > > > multipart > > > > > > > > > > > > > > > > > > > > > > operations, negative scope-boundary > > > > > tests, > > > > > > > our > > > > > > > > > > > internal > > > > > > > > > > > > > > > Python > > > > > > > > > > > > > > > > > > client > > > > > > > > > > > > > > > > > > > > > > regression matrix, and credential > > > > > > > > > rotation/recovery > > > > > > > > > > > > > > > exercises. > > > > > > > > > > > > > > > > > The > > > > > > > > > > > > > > > > > > > > > upstream > > > > > > > > > > > > > > > > > > > > > > CI workflow has passed on the fork. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > The change also encounters the > > > > existing > > > > > > > > > > assumption > > > > > > > > > > > > > that a > > > > > > > > > > > > > > > > > FileIO > > > > > > > > > > > > > > > > > > > > > > implementation identifies exactly > > one > > > > > > storage > > > > > > > > > type: > > > > > > > > > > > > both > > > > > > > > > > > > > S3 > > > > > > > > > > > > > > > and > > > > > > > > > > > > > > > > > R2 > > > > > > > > > > > > > > > > > > > use > > > > > > > > > > > > > > > > > > > > > > S3FileIO. This overlaps with #4486 > > > [3]. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I’d particularly appreciate > > > feedback > > > > > on: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 1. > > > > > > > > > > > > > > > > > > > > > > > Should R2 have its own storage > > > > > > > > > > configuration/type, > > > > > > > > > > > or > > > > > > > > > > > > > > > should > > > > > > > > > > > > > > > > > > > non-STS > > > > > > > > > > > > > > > > > > > > > > vending be modeled within the > > > existing > > > > S3 > > > > > > > > > > > integration? > > > > > > > > > > > > > > > > > > > > > > > 2. > > > > > > > > > > > > > > > > > > > > > > > Does the local-signing approach > > fit > > > > > > > Polaris’s > > > > > > > > > > > > > > > > > credential-vending > > > > > > > > > > > > > > > > > > > > model? > > > > > > > > > > > > > > > > > > > > > > Any additional constraints or > > > > validations > > > > > > > > > > necessary? > > > > > > > > > > > > > > > > > > > > > > > 3. > > > > > > > > > > > > > > > > > > > > > > > Would you prefer the shared > > FileIO > > > > > > > validation > > > > > > > > > > > change > > > > > > > > > > > > to > > > > > > > > > > > > > > > land > > > > > > > > > > > > > > > > > > > > > separately, > > > > > > > > > > > > > > > > > > > > > > and how should we sequence the > > > > management > > > > > > > API and > > > > > > > > > > R2 > > > > > > > > > > > > > > > > integration > > > > > > > > > > > > > > > > > > > > changes? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Happy to adapt the design and > > > split > > > > > the > > > > > > > > > > > > contribution > > > > > > > > > > > > > > into > > > > > > > > > > > > > > > > > more > > > > > > > > > > > > > > > > > > > > > focused > > > > > > > > > > > > > > > > > > > > > > PRs. Wanted to start the discussion > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > > > > > > > > Austen > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > [1] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://developers.cloudflare.com/r2/api/s3/temporary-credentials/ > > > > > > > > > > > > > > > > > > > > > > > [2] > > > > > > > > > > > https://github.com/deepdishgary/polaris/pull/1 > > > > > > > > > > > > > > > > > > > > > > > [3] > > > > > > > > > > > https://github.com/apache/polaris/issues/4486 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Get Outlook for Mac< > > > > > > > > > > > https://aka.ms/GetOutlookForMac> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > This electronic mail message and > > > any > > > > > > > attached > > > > > > > > > > files > > > > > > > > > > > > > > contain > > > > > > > > > > > > > > > > > > > > information > > > > > > > > > > > > > > > > > > > > > > intended for the exclusive use of > > the > > > > > > > individual > > > > > > > > > or > > > > > > > > > > > > > entity > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > whom > > > > > > > > > > > > > > > > > > it > > > > > > > > > > > > > > > > > > > > is > > > > > > > > > > > > > > > > > > > > > > addressed and may contain > > information > > > > > that > > > > > > is > > > > > > > > > > > > > proprietary, > > > > > > > > > > > > > > > > > > > confidential > > > > > > > > > > > > > > > > > > > > > > and/or exempt from disclosure under > > > > > > > applicable > > > > > > > > > law. > > > > > > > > > > > If > > > > > > > > > > > > > you > > > > > > > > > > > > > > > are > > > > > > > > > > > > > > > > > not > > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > > intended recipient, you are hereby > > > > > notified > > > > > > > that > > > > > > > > > > any > > > > > > > > > > > > > > viewing, > > > > > > > > > > > > > > > > > > > copying, > > > > > > > > > > > > > > > > > > > > > > disclosure or distribution of this > > > > > > > information > > > > > > > > > may > > > > > > > > > > be > > > > > > > > > > > > > > subject > > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > > > legal > > > > > > > > > > > > > > > > > > > > > > restriction or sanction. Please > > > notify > > > > > the > > > > > > > > > sender, > > > > > > > > > > by > > > > > > > > > > > > > > > > electronic > > > > > > > > > > > > > > > > > > mail > > > > > > > > > > > > > > > > > > > > or > > > > > > > > > > > > > > > > > > > > > > telephone, of any unintended > > > recipients > > > > > and > > > > > > > > > delete > > > > > > > > > > > the > > > > > > > > > > > > > > > original > > > > > > > > > > > > > > > > > > > message > > > > > > > > > > > > > > > > > > > > > > without making any copies. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > This electronic mail message and any > > > attached > > > > > > files > > > > > > > > > > contain > > > > > > > > > > > > > > > > information > > > > > > > > > > > > > > > > > > > intended for the exclusive use of the > > > > > individual > > > > > > or > > > > > > > > > > entity > > > > > > > > > > > to > > > > > > > > > > > > > > whom > > > > > > > > > > > > > > > it > > > > > > > > > > > > > > > > > is > > > > > > > > > > > > > > > > > > > addressed and may contain information > > that > > > is > > > > > > > > > > proprietary, > > > > > > > > > > > > > > > > confidential > > > > > > > > > > > > > > > > > > > and/or exempt from disclosure under > > > > applicable > > > > > > > law. If > > > > > > > > > > you > > > > > > > > > > > > are > > > > > > > > > > > > > > not > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > intended recipient, you are hereby > > notified > > > > > that > > > > > > > any > > > > > > > > > > > viewing, > > > > > > > > > > > > > > > > copying, > > > > > > > > > > > > > > > > > > > disclosure or distribution of this > > > > information > > > > > > may > > > > > > > be > > > > > > > > > > > subject > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > legal > > > > > > > > > > > > > > > > > > > restriction or sanction. Please notify > > the > > > > > > sender, > > > > > > > by > > > > > > > > > > > > > electronic > > > > > > > > > > > > > > > mail > > > > > > > > > > > > > > > > > or > > > > > > > > > > > > > > > > > > > telephone, of any unintended recipients > > and > > > > > > delete > > > > > > > the > > > > > > > > > > > > original > > > > > > > > > > > > > > > > message > > > > > > > > > > > > > > > > > > > without making any copies. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > This electronic mail message and any attached > > > files > > > > > > > contain > > > > > > > > > > > > > information > > > > > > > > > > > > > > > > intended for the exclusive use of the > > individual > > > or > > > > > > > entity to > > > > > > > > > > > whom > > > > > > > > > > > > it > > > > > > > > > > > > > > is > > > > > > > > > > > > > > > > addressed and may contain information that is > > > > > > > proprietary, > > > > > > > > > > > > > confidential > > > > > > > > > > > > > > > > and/or exempt from disclosure under applicable > > > law. > > > > > If > > > > > > > you > > > > > > > > > are > > > > > > > > > > > not > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > intended recipient, you are hereby notified > > that > > > > any > > > > > > > viewing, > > > > > > > > > > > > > copying, > > > > > > > > > > > > > > > > disclosure or distribution of this information > > > may > > > > be > > > > > > > subject > > > > > > > > > > to > > > > > > > > > > > > > legal > > > > > > > > > > > > > > > > restriction or sanction. Please notify the > > > sender, > > > > by > > > > > > > > > > electronic > > > > > > > > > > > > mail > > > > > > > > > > > > > > or > > > > > > > > > > > > > > > > telephone, of any unintended recipients and > > > delete > > > > > the > > > > > > > > > original > > > > > > > > > > > > > message > > > > > > > > > > > > > > > > without making any copies. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > This electronic mail message and any attached > > files > > > > > > contain > > > > > > > > > > > > information > > > > > > > > > > > > > > > intended for the exclusive use of the individual > > or > > > > > > entity > > > > > > > to > > > > > > > > > > whom > > > > > > > > > > > it > > > > > > > > > > > > > is > > > > > > > > > > > > > > > addressed and may contain information that is > > > > > > proprietary, > > > > > > > > > > > > confidential > > > > > > > > > > > > > > > and/or exempt from disclosure under applicable > > law. > > > > If > > > > > > you > > > > > > > are > > > > > > > > > > not > > > > > > > > > > > > the > > > > > > > > > > > > > > > intended recipient, you are hereby notified that > > > any > > > > > > > viewing, > > > > > > > > > > > > copying, > > > > > > > > > > > > > > > disclosure or distribution of this information > > may > > > be > > > > > > > subject > > > > > > > > > to > > > > > > > > > > > > legal > > > > > > > > > > > > > > > restriction or sanction. Please notify the > > sender, > > > by > > > > > > > > > electronic > > > > > > > > > > > mail > > > > > > > > > > > > > or > > > > > > > > > > > > > > > telephone, of any unintended recipients and > > delete > > > > the > > > > > > > original > > > > > > > > > > > > message > > > > > > > > > > > > > > > without making any copies. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > This electronic mail message and any attached files > > > > contain > > > > > > > > > > information > > > > > > > > > > > > > intended for the exclusive use of the individual or > > > > entity > > > > > to > > > > > > > whom > > > > > > > > > it > > > > > > > > > > > is > > > > > > > > > > > > > addressed and may contain information that is > > > > proprietary, > > > > > > > > > > confidential > > > > > > > > > > > > > and/or exempt from disclosure under applicable law. > > If > > > > you > > > > > > are > > > > > > > not > > > > > > > > > > the > > > > > > > > > > > > > intended recipient, you are hereby notified that any > > > > > viewing, > > > > > > > > > > copying, > > > > > > > > > > > > > disclosure or distribution of this information may be > > > > > subject > > > > > > > to > > > > > > > > > > legal > > > > > > > > > > > > > restriction or sanction. Please notify the sender, by > > > > > > > electronic > > > > > > > > > mail > > > > > > > > > > > or > > > > > > > > > > > > > telephone, of any unintended recipients and delete > > the > > > > > > original > > > > > > > > > > message > > > > > > > > > > > > > without making any copies. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > This electronic mail message and any attached files contain > > information > > > > > intended for the exclusive use of the individual or entity to whom it > > > is > > > > > addressed and may contain information that is proprietary, > > confidential > > > > > and/or exempt from disclosure under applicable law. If you are not > > the > > > > > intended recipient, you are hereby notified that any viewing, > > copying, > > > > > disclosure or distribution of this information may be subject to > > legal > > > > > restriction or sanction. Please notify the sender, by electronic mail > > > or > > > > > telephone, of any unintended recipients and delete the original > > message > > > > > without making any copies. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > This electronic mail message and any attached files contain information > > > intended for the exclusive use of the individual or entity to whom it is > > > addressed and may contain information that is proprietary, confidential > > > and/or exempt from disclosure under applicable law. If you are not the > > > intended recipient, you are hereby notified that any viewing, copying, > > > disclosure or distribution of this information may be subject to legal > > > restriction or sanction. Please notify the sender, by electronic mail or > > > telephone, of any unintended recipients and delete the original message > > > without making any copies. > > > > > > > > > > > > > > > > > This electronic mail message and any attached files contain information > > intended for the exclusive use of the individual or entity to whom it is > > addressed and may contain information that is proprietary, confidential > > and/or exempt from disclosure under applicable law. If you are not the > > intended recipient, you are hereby notified that any viewing, copying, > > disclosure or distribution of this information may be subject to legal > > restriction or sanction. Please notify the sender, by electronic mail or > > telephone, of any unintended recipients and delete the original message > > without making any copies. > > This electronic mail message and any attached files contain information intended for the exclusive use of the individual or entity to whom it is addressed and may contain information that is proprietary, confidential and/or exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any viewing, copying, disclosure or distribution of this information may be subject to legal restriction or sanction. Please notify the sender, by electronic mail or telephone, of any unintended recipients and delete the original message without making any copies.
