Hi all, I'm having trouble understanding how AWS system-wide credentials work when subscoping or STS is unavailable. My apologies if this has been asked previously, but I would greatly appreciate it if someone could provide an explanation.
Currently, Polaris honors AWS system-wide credentials (set via polaris.storage.aws.access-key and polaris.storage.aws.secret-key) only when credentials subscoping is enabled and STS is available. In this scenario, Polaris generates subscoped credentials using these static credentials for every STS request [1]. However, if either credentials subscoping or STS is unavailable, these system-wide credentials are ignored, afaict. The core problem, in my opinion, is that the current code mixes credentials subscoping (for the server itself) and credentials vending (for clients). While it's appropriate not to vend these long-lived credentials directly to clients, I believe they should be used when the server itself needs to access the remote storage. This leads to issues in two specific S3 setups: 1) System-wide credentials with credentials subscoping OFF (STS status is irrelevant). 2) System-wide credentials with credentials subscoping ON, but STS marked unavailable. In both cases, the generated AccessConfig doesn't contain credentials, which means that the FileIO instance that Polaris creates will rely on the default provider chain for credentials. I wonder if the AccessConfig should instead contain the keys s3.access-key-id and s3.secret-access-key, populated with the static, system-wide credentials? If I were a user using an S3-like service without STS, I would have expected that those static credentials would be used for FileIO instance creation (and also, one day, for S3 remote signing). To address this, I was thinking of adding a dedicated method to AccessConfigProvider. Something like getClientAccessConfig() vs getServerAccessConfig(). The latter would return a specialized AccessConfig for server-side usage, ensuring that if static credentials are present, these are used whenever subscoping is off or STS is unavailable. What do you all think? Thanks, Alex [1]: https://github.com/apache/polaris/blob/f97c5eb50016489129575aab62d5efb3efb7552e/polaris-core/src/main/java/org/apache/polaris/core/storage/aws/AwsCredentialsStorageIntegration.java#L99-L100
