I added a separate STS endpoint property to https://github.com/apache/polaris/pull/1913
Cheers, Dmitri. On Wed, Jun 25, 2025 at 7:59 PM Dmitri Bourlatchkov <di...@apache.org> wrote: > Thanks for the feedback, Adnan! > > I was initially thinking of deferring separating STS and S3 endpoints to a > later PR, but given interest, I'll add it to the current PR to reduce the > number of API changes over time. > > Cheers, > Dmitri. > > On Wed, Jun 25, 2025 at 1:28 PM Adnan Hemani > <adnan.hem...@snowflake.com.invalid> wrote: > >> +1 on overall idea >> >> Comment however on keeping the same endpoint for STS and S3 - there may be >> a customer use case (I have seen this before for similar use cases) where >> the S3 bucket is not in the same region as the STS endpoint that has been >> exposed to the Polaris server. In that case, you really need two separate >> endpoints (example below). But I don't think this necessarily blocks this >> idea from being merged today and then enriched later. >> >> Example: >> * Polaris is deployed in us-west-1 in a EC2 instance that is within a >> private subnet, which is given access only to STS' us-west-1 regional >> endpoint through AWS configurations. >> * User has S3 data in both us-west-1 and us-east-1. >> * Under this proposal, user can now access data in us-west-1, but when >> trying to access data in us-east-1, STS will not respond due to no network >> routing tables for the us-east-1 regional endpoint. >> * Important thing to remember is that STS token generated at any regional >> endpoint can be used at any other regional endpoint. So, in theory, user >> could still use us-west-1 STS to generate credentials to use for S3 data >> in >> us-east-1 and it would've been ok - but this proposal does not allow for >> this. >> >> Best, >> Adnan Hemani >> >> On Tue, Jun 24, 2025 at 9:23 PM Dmitri Bourlatchkov <di...@apache.org> >> wrote: >> >> > Hi All, >> > >> > I propose to add an `endpoint` optional parameter to >> AwsStorageConfigInfo >> > in PR [1913]. >> > >> > The main idea is to support non-AWS S3 implementations for [1530]. >> > >> > Existing classes related to supporting S3 Polaris are coded to the AWS >> SDK, >> > which supports setting STS endpoints. Therefore, it seems natural to >> allow >> > users to optionally define a specific endpoint for their catalogs. >> > >> > This change is backward-compatible with existing clients and deployed >> > catalogs. >> > >> > When an endpoint is defined it will be used for both STS and S3 requests >> > inside Polaris and will be used to populate the "s3.endpoint" >> properties in >> > REST Catalog clients. >> > >> > Thoughts? >> > >> > [1530] https://github.com/apache/polaris/issues/1530 >> > [1913] https://github.com/apache/polaris/pull/1913 >> > >> > Thanks, >> > Dmitri. >> > >> >
