Thibaut Cazenave created CAMEL-26367:
----------------------------------------
Summary: camel-aws: AwsClientBuilderUtil shares the
DefaultCredentialsProvider singleton across clients; closing one endpoint
breaks all others ("Connection pool shut down")
Key: CAMEL-26367
URL: https://issues.apache.org/jira/browse/CAMEL-26367
Project: Camel
Issue Type: Bug
Components: camel-aws2, camel-aws2-s3, camel-aws2-sqs
Affects Versions: 4.22.0, 4.18.0
Reporter: Thibaut Cazenave
`AwsClientBuilderUtil.resolveCredentialsProvider()` returns
`DefaultCredentialsProvider.create()` when
`useDefaultCredentialsProvider=true`. In AWS SDK v2, `create()` returns a
JVM-wide singleton. The SDK has deprecated it for this reason: "can cause
issues if one client closes the provider while others are still using it. Use
builder().build() to create independent instances".
`Sqs2Endpoint.doStop()` calls `client.close()`, which closes the shared
provider and its provider chain. With IRSA (EKS / OpenShift web identity), that
includes the STS client used by `WebIdentityTokenFileCredentialsProvider`.
Every other AWS client in the same JVM keeps its cached credentials for a
while. When those expire, every refresh fails, and `Sqs2Consumer` logs and
retries in a tight loop:
```
ERROR Sqs2Consumer - Error while polling
java.lang.IllegalStateException: Connection pool shut down
at
org.apache.http.impl.conn.PoolingHttpClientConnectionManager.requestConnection(...)
...
at
software.amazon.awssdk.services.sts.DefaultStsClient.assumeRoleWithWebIdentity(DefaultStsClient.java:778)
at
software.amazon.awssdk.services.sts.auth.StsAssumeRoleWithWebIdentityCredentialsProvider.getUpdatedCredentials(...)
at
software.amazon.awssdk.auth.credentials.WebIdentityTokenFileCredentialsProvider.resolveCredentials(...)
at
software.amazon.awssdk.auth.credentials.DefaultCredentialsProvider.resolveCredentials(DefaultCredentialsProvider.java:134)
at
software.amazon.awssdk.services.sqs.DefaultSqsClient.receiveMessage(DefaultSqsClient.java:3352)
at
org.apache.camel.component.aws2.sqs.Sqs2Consumer$PollingTask.poll(Sqs2Consumer.java:645)
```
The same happens with the Apache 5 client in 4.22.0, through
`software.amazon.awssdk.http.apache5`.
**Environment:** Camel Kafka Connector `camel-aws-sqs-source-kafka-connector`
4.18.0 and 4.22.0 on Kafka Connect 4.3.1 (Strimzi). About 37 SQS source
connectors with `useDefaultCredentialsProvider=true`, spread over 3 workers,
using IRSA.
**Steps to reproduce**
1. Run several aws2-sqs consumers in one JVM, each in its own CamelContext (as
Camel Kafka Connector does per task), with `useDefaultCredentialsProvider=true`
and web-identity credentials.
2. Stop one of them, for example a Kafka Connect rebalance or task restart, or
stopping one route or endpoint.
3. Wait for the cached STS credentials to expire (about 1 hour by default).
4. Every remaining consumer in that JVM fails on every poll with `Connection
pool shut down`, permanently.
**Observed**
On each worker that had a task revoked during a rolling restart, all of its
tasks failed (12/12 and 13/13). Workers that never had a task stopped were
unaffected (12/12 healthy). Kafka Connect still reports every connector as
RUNNING.
**Regression:** in 4.11.0, `Sqs2ClientIAMOptimized` did not set a credentials
provider, so each client got its own SDK-managed default provider. The shared
`AwsClientBuilderUtil` introduced the singleton, and it also affects the other
camel-aws components that use it.
**Suggested fix:** in `AwsClientBuilderUtil.resolveCredentialsProvider()`, use
`DefaultCredentialsProvider.builder().build()`, or leave the provider unset so
the SDK creates and owns one per client.
**Related**
* CAMEL-25909 (camel-aws2-kinesis) and CAMEL-25907 (camel-elasticsearch /
camel-opensearch): same pattern, where stopping one endpoint closes a client
that other endpoints still use.
* CAMEL-19314: earlier "Connection pool shut down" bug in camel-aws (health
checks closing the client), resolved in 4.0.0.
* aws/aws-sdk-java-v2#4386: same symptom with web identity credentials. It
works for an hour, then every request fails with "Connection pool shut down",
because `DefaultCredentialsProvider.create()` is a global instance and the
first `close()` breaks every other user once the token refreshes. This is why
the SDK deprecated `create()`.
* aws/aws-sdk-java-v2#4221: same setup (web identity plus STS), closed with
"something outside the SDK is closing the STS client".
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
