Kousuke Saruta created SPARK-57899:
--------------------------------------
Summary: Implement SparkOidcAwsCredentialsProvider
Key: SPARK-57899
URL: https://issues.apache.org/jira/browse/SPARK-57899
Project: Spark
Issue Type: Sub-task
Components: Spark Core
Affects Versions: 4.3.0
Reporter: Kousuke Saruta
This sub-task is part of the umbrella
[SPARK-57703|https://issues.apache.org/jira/browse/SPARK-57703] (OIDC
Credential Propagation).
h2. Problem
Executor-side S3A FileSystem needs to pick up refreshed credentials from the
credential store without requiring FileSystem cache invalidation.
h2. Goal
Implement a dynamic AWS credentials provider that reads from the executor
credential store on every call.
h2. Scope
* Implement {{software.amazon.awssdk.auth.credentials.AwsCredentialsProvider}}
in {{connector/credential-aws}}
* On each {{resolveCredentials()}} call, read current {{ServiceCredential}}
from executor credential store
* Never cache credentials internally (always return latest from store)
* Auto-enabled when {{spark.security.credentials.enabled=true}} and user has
not explicitly configured {{fs.s3a.aws.credentials.provider}}
h2. Acceptance criteria
* After credential refresh (store update), the next {{resolveCredentials()}}
call returns new credentials.
* No stale credentials are returned after a refresh.
* User-provided {{fs.s3a.aws.credentials.provider}} is not overwritten.
* Unit tests verify dynamic behavior with simulated credential store updates.
Reference: [SPIP design document Appendix
B|https://docs.google.com/document/d/1usJKncCPMiyFUg7aIdpZ0HQsklXIHow_sU_6dfFMjN0/edit].
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
