This is an automated email from the ASF dual-hosted git repository. github-merge-queue[bot] pushed a commit to branch gh-readonly-queue/dev/pr-11423-94e8490971b969e9c60a280cd686edebd34d7209 in repository https://gitbox.apache.org/repos/asf/seatunnel.git
commit 8ceebc754c6f4b63efb925b56a29cdcafc06b5b6 Author: lifeng <[email protected]> AuthorDate: Sun Oct 11 09:46:00 2026 +0000 [Docs] Clarify S3File credential providers for container environments (#11423) Co-authored-by: David Zollo <[email protected]> Co-authored-by: Claude Sonnet 5.5 <[email protected]> --- docs/en/engines/zeta/checkpoint-storage.md | 32 +++++++++++++++++++++++++++--- docs/zh/engines/zeta/checkpoint-storage.md | 32 +++++++++++++++++++++++++++--- 2 files changed, 58 insertions(+), 6 deletions(-) diff --git a/docs/en/engines/zeta/checkpoint-storage.md b/docs/en/engines/zeta/checkpoint-storage.md index f20db8a6b6..b3db813485 100644 --- a/docs/en/engines/zeta/checkpoint-storage.md +++ b/docs/en/engines/zeta/checkpoint-storage.md @@ -180,7 +180,7 @@ seatunnel: plugin-config: namespace: # checkpoint storage parent path, the default value is /seatunnel/checkpoint/ storage.type: s3 - s3.bucket: your-bucket + s3.bucket: s3a://your-bucket fs.s3a.access.key: your-access-key fs.s3a.secret.key: your-secret-key fs.s3a.aws.credentials.provider: org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider @@ -204,11 +204,37 @@ seatunnel: plugin-config: namespace: # checkpoint storage parent path, the default value is /seatunnel/checkpoint/ storage.type: s3 - s3.bucket: your-bucket + s3.bucket: s3a://your-bucket fs.s3a.endpoint: your-endpoint - fs.s3a.aws.credentials.provider: org.apache.hadoop.fs.s3a.InstanceProfileCredentialsProvider + fs.s3a.aws.credentials.provider: com.amazonaws.auth.InstanceProfileCredentialsProvider ``` + +Checkpoint storage differs from the S3File source and sink: values under `plugin-config` are passed directly to Hadoop configuration and are not limited by S3File option validation. With the AWS SDK v1 bundled in the current distribution, `com.amazonaws.auth.DefaultAWSCredentialsProviderChain` can read environment, profile, ECS container, and EC2 instance profile credentials. For an ECS task role, you can also select the container provider explicitly: + +```yaml +seatunnel: + engine: + checkpoint: + interval: 6000 + timeout: 7000 + storage: + type: hdfs + max-retained: 3 + plugin-config: + namespace: # checkpoint storage parent path, the default value is /seatunnel/checkpoint/ + storage.type: s3 + s3.bucket: s3a://your-bucket + fs.s3a.endpoint: your-endpoint + fs.s3a.aws.credentials.provider: com.amazonaws.auth.ContainerCredentialsProvider +``` + +For Kubernetes or EKS nodes backed by EC2, use `com.amazonaws.auth.InstanceProfileCredentialsProvider` as shown above and grant the node role only the required bucket and prefix permissions. + +The AWS SDK v1 bundled with the current checkpoint-storage dependencies does not contain `WebIdentityTokenCredentialsProvider`, so EKS IRSA is not supported. Do not configure an IRSA provider unless the runtime dependencies have been upgraded and validated together. + +If `Factory initialize failed` or `ClassNotFoundException` appears, verify the provider class name and confirm that every master and worker accessing checkpoint storage has the required Hadoop/AWS jars. + **Container environments**: Checkpoint storage passes `fs.s3a.*` configuration keys directly to Hadoop without any connector-level enum restriction, so any S3A credential provider class available on the classpath can be used. This includes container-oriented providers such as `com.amazonaws.auth.ContainerCredentialsProvider` (ECS task role) and `com.amazonaws.auth.DefaultAWSCredentialsProviderChain`. For EKS deployments, the EC2 node instance role is recommended. EKS IRSA (`WebIdentityTo [...] If you want to use Minio that supports the S3 protocol as checkpoint storage, you should configure it this way: diff --git a/docs/zh/engines/zeta/checkpoint-storage.md b/docs/zh/engines/zeta/checkpoint-storage.md index 2cc3852ec3..4e87f108a4 100644 --- a/docs/zh/engines/zeta/checkpoint-storage.md +++ b/docs/zh/engines/zeta/checkpoint-storage.md @@ -178,7 +178,7 @@ seatunnel: plugin-config: namespace: # 检查点存储父路径，默认值为/seatunnel/checkpoint/ storage.type: s3 - s3.bucket: your-bucket + s3.bucket: s3a://your-bucket fs.s3a.access.key: your-access-key fs.s3a.secret.key: your-secret-key fs.s3a.aws.credentials.provider: org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider @@ -202,11 +202,37 @@ seatunnel: plugin-config: namespace: # 检查点存储父路径，默认值为/seatunnel/checkpoint/ storage.type: s3 - s3.bucket: your-bucket + s3.bucket: s3a://your-bucket fs.s3a.endpoint: your-endpoint - fs.s3a.aws.credentials.provider: org.apache.hadoop.fs.s3a.InstanceProfileCredentialsProvider + fs.s3a.aws.credentials.provider: com.amazonaws.auth.InstanceProfileCredentialsProvider ``` + +Checkpoint 存储与 S3File source/sink 不同：`plugin-config` 下的值会直接传入 Hadoop configuration，不受 S3File 选项校验限制。使用当前发行包内置的 AWS SDK v1 类时，`com.amazonaws.auth.DefaultAWSCredentialsProviderChain` 可以读取环境变量、profile、ECS container 和 EC2 instance profile 凭据。对于 ECS task role，也可以显式选择 container provider： + +```yaml +seatunnel: + engine: + checkpoint: + interval: 6000 + timeout: 7000 + storage: + type: hdfs + max-retained: 3 + plugin-config: + namespace: # 检查点存储父路径，默认值为/seatunnel/checkpoint/ + storage.type: s3 + s3.bucket: s3a://your-bucket + fs.s3a.endpoint: your-endpoint + fs.s3a.aws.credentials.provider: com.amazonaws.auth.ContainerCredentialsProvider +``` + +对于基于 EC2 的 Kubernetes 或 EKS 节点，可按上面的示例使用 `com.amazonaws.auth.InstanceProfileCredentialsProvider`，并仅向节点角色授予所需 bucket/prefix 的最小权限。 + +当前 checkpoint-storage 依赖内置的 AWS SDK v1 版本不包含 `WebIdentityTokenCredentialsProvider`，因此不支持 EKS IRSA。除非运行时依赖已经整体升级并完成验证，否则不要配置 IRSA provider。 + +如果遇到 `Factory initialize failed` 或 `ClassNotFoundException`，请检查 provider 类名，并确认所有访问 checkpoint 存储的 master/worker 都已加载所需的 Hadoop/AWS jar。 + **容器环境**：检查点存储将 `fs.s3a.*` 配置键直接传递给 Hadoop，没有连接器级别的枚举限制，因此可以使用 classpath 上任何可用的 S3A 凭据提供程序类。这包括面向容器的提供程序，如 `com.amazonaws.auth.ContainerCredentialsProvider`（ECS 任务角色）和 `com.amazonaws.auth.DefaultAWSCredentialsProviderChain`。对于 EKS 部署，推荐使用 EC2 节点实例角色。EKS IRSA（`WebIdentityTokenCredentialsProvider`）在捆绑的 AWS SDK v1.x（1.11.271）中不可用，需要在所有节点的 `${SEATUNNEL_HOME}/lib` 中添加较新的 AWS SDK v1.x JAR。 有关Hadoop Credential Provider API的更多信息，请参见: [Credential Provider API](https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/CredentialProviderAPI.html).
