[
https://issues.apache.org/jira/browse/CAMEL-26367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrea Cosentino updated CAMEL-26367:
-------------------------------------
Fix Version/s: 4.18.6
> camel-aws: AwsClientBuilderUtil shares the DefaultCredentialsProvider
> singleton across clients; closing one endpoint breaks all others ("Connection
> pool shut down")
> --------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: CAMEL-26367
> URL: https://issues.apache.org/jira/browse/CAMEL-26367
> Project: Camel
> Issue Type: Bug
> Components: camel-aws2, camel-aws2-s3, camel-aws2-sqs
> Affects Versions: 4.18.0, 4.22.0
> Reporter: Thibaut Cazenave
> Assignee: Andrea Cosentino
> Priority: Major
> Fix For: 4.18.6, 4.22.3, 4.24.0
>
>
> `AwsClientBuilderUtil.resolveCredentialsProvider()` returns
> `DefaultCredentialsProvider.create()` when
> `useDefaultCredentialsProvider=true`. In AWS SDK v2, `create()` returns a
> JVM-wide singleton. The SDK has deprecated it for this reason: "can cause
> issues if one client closes the provider while others are still using it. Use
> builder().build() to create independent instances".
> `Sqs2Endpoint.doStop()` calls `client.close()`, which closes the shared
> provider and its provider chain. With IRSA (EKS / OpenShift web identity),
> that includes the STS client used by
> `WebIdentityTokenFileCredentialsProvider`. Every other AWS client in the same
> JVM keeps its cached credentials for a while. When those expire, every
> refresh fails, and `Sqs2Consumer` logs and retries in a tight loop:
> ```
> ERROR Sqs2Consumer - Error while polling
> java.lang.IllegalStateException: Connection pool shut down
> at
> org.apache.http.impl.conn.PoolingHttpClientConnectionManager.requestConnection(...)
> ...
> at
> software.amazon.awssdk.services.sts.DefaultStsClient.assumeRoleWithWebIdentity(DefaultStsClient.java:778)
> at
> software.amazon.awssdk.services.sts.auth.StsAssumeRoleWithWebIdentityCredentialsProvider.getUpdatedCredentials(...)
> at
> software.amazon.awssdk.auth.credentials.WebIdentityTokenFileCredentialsProvider.resolveCredentials(...)
> at
> software.amazon.awssdk.auth.credentials.DefaultCredentialsProvider.resolveCredentials(DefaultCredentialsProvider.java:134)
> at
> software.amazon.awssdk.services.sqs.DefaultSqsClient.receiveMessage(DefaultSqsClient.java:3352)
> at
> org.apache.camel.component.aws2.sqs.Sqs2Consumer$PollingTask.poll(Sqs2Consumer.java:645)
> ```
> The same happens with the Apache 5 client in 4.22.0, through
> `software.amazon.awssdk.http.apache5`.
> **Environment:** Camel Kafka Connector `camel-aws-sqs-source-kafka-connector`
> 4.18.0 and 4.22.0 on Kafka Connect 4.3.1 (Strimzi). About 37 SQS source
> connectors with `useDefaultCredentialsProvider=true`, spread over 3 workers,
> using IRSA.
> **Steps to reproduce**
> 1. Run several aws2-sqs consumers in one JVM, each in its own CamelContext
> (as Camel Kafka Connector does per task), with
> `useDefaultCredentialsProvider=true` and web-identity credentials.
> 2. Stop one of them, for example a Kafka Connect rebalance or task restart,
> or stopping one route or endpoint.
> 3. Wait for the cached STS credentials to expire (about 1 hour by default).
> 4. Every remaining consumer in that JVM fails on every poll with `Connection
> pool shut down`, permanently.
> **Observed**
> On each worker that had a task revoked during a rolling restart, all of its
> tasks failed (12/12 and 13/13). Workers that never had a task stopped were
> unaffected (12/12 healthy). Kafka Connect still reports every connector as
> RUNNING.
> **Regression:** in 4.11.0, `Sqs2ClientIAMOptimized` did not set a credentials
> provider, so each client got its own SDK-managed default provider. The shared
> `AwsClientBuilderUtil` introduced the singleton, and it also affects the
> other camel-aws components that use it.
> **Suggested fix:** in `AwsClientBuilderUtil.resolveCredentialsProvider()`,
> use `DefaultCredentialsProvider.builder().build()`, or leave the provider
> unset so the SDK creates and owns one per client.
> **Related**
> * CAMEL-25909 (camel-aws2-kinesis) and CAMEL-25907 (camel-elasticsearch /
> camel-opensearch): same pattern, where stopping one endpoint closes a client
> that other endpoints still use.
> * CAMEL-19314: earlier "Connection pool shut down" bug in camel-aws (health
> checks closing the client), resolved in 4.0.0.
> * aws/aws-sdk-java-v2#4386: same symptom with web identity credentials. It
> works for an hour, then every request fails with "Connection pool shut down",
> because `DefaultCredentialsProvider.create()` is a global instance and the
> first `close()` breaks every other user once the token refreshes. This is why
> the SDK deprecated `create()`.
> * aws/aws-sdk-java-v2#4221: same setup (web identity plus STS), closed with
> "something outside the SDK is closing the STS client".
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
