Based on the stacktrace, this looks like an issue described here: https://issues.apache.org/jira/browse/FLINK-32964 Is your configuration similar to the one described in the ticket? If so, you can work around this issue by explicitly specifying the credentials provider for connector, by doing so avoiding using DefaultCredentialsProvider (AUTO).
Caused by: java.lang.IllegalStateException: Connection pool shut down ... at org.apache.flink.kinesis.shaded.software.amazon.awssdk.auth.credentials. *WebIdentityTokenFileCredentialsProvider* .resolveCredentials(WebIdentityTokenFileCredentialsProvider.java: 143) ~[?:?] ... at org.apache.flink.kinesis.shaded.software.amazon.awssdk.auth.credentials. *DefaultCredentialsProvider* .resolveCredentials(DefaultCredentialsProvider.java: 128) ~[?:?] On Thu, 29 Feb 2024 at 02:24, Xiaolong Wang <xiaolong.w...@smartnews.com.invalid> wrote: > Sorry, I just attached a wrong file. Let me paste the error log: > > java.lang.RuntimeException: Maximum retries exceeded for SubscribeToShard. > Failed 10 times. > at > > org.apache.flink.streaming.connectors.kinesis.internals.publisher.fanout.FanOutRecordPublisher.runWithBackoff(FanOutRecordPublisher.java: > 211) ~[?:?] > at > > org.apache.flink.streaming.connectors.kinesis.internals.publisher.fanout.FanOutRecordPublisher.run(FanOutRecordPublisher.java: > 130) ~[?:?] > at > > org.apache.flink.streaming.connectors.kinesis.internals.ShardConsumer.run(ShardConsumer.java: > 114) ~[?:?] > at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) > ~[?:? > ] > at java.util.concurrent.FutureTask.run(Unknown Source) ~[?:?] > at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[?:?] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) > ~[?:?] > at java.lang.Thread.run(Unknown Source) ~[?:?] > Caused by: java.lang.IllegalStateException: Connection pool shut down > at > > org.apache.flink.kinesis.shaded.org.apache.http.util.Asserts.check(Asserts.java: > 34) ~[?:?] > at > > org.apache.flink.kinesis.shaded.org.apache.http.impl.conn.PoolingHttpClientConnectionManager.requestConnection(PoolingHttpClientConnectionManager.java: > 269) ~[?:?] > at > > org.apache.flink.kinesis.shaded.software.amazon.awssdk.http.apache.internal.conn.ClientConnectionManagerFactory > $ > > DelegatingHttpClientConnectionManager.requestConnection(ClientConnectionManagerFactory.java: > 75) ~[?:?] > at > > org.apache.flink.kinesis.shaded.software.amazon.awssdk.http.apache.internal.conn.ClientConnectionManagerFactory > $ > > InstrumentedHttpClientConnectionManager.requestConnection(ClientConnectionManagerFactory.java: > 57) ~[?:?] > at > > org.apache.flink.kinesis.shaded.org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java: > 176) ~[?:?] > at > > org.apache.flink.kinesis.shaded.org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java: > 186) ~[?:?] > at > > org.apache.flink.kinesis.shaded.org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java: > 185) ~[?:?] > at > > org.apache.flink.kinesis.shaded.org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java: > 83) ~[?:?] > at > > org.apache.flink.kinesis.shaded.org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java: > 56) ~[?:?] > at > > org.apache.flink.kinesis.shaded.software.amazon.awssdk.http.apache.internal.impl.ApacheSdkHttpClient.execute(ApacheSdkHttpClient.java: > 72) ~[?:?] > at > > org.apache.flink.kinesis.shaded.software.amazon.awssdk.http.apache.ApacheHttpClient.execute(ApacheHttpClient.java: > 254) ~[?:?] > at > > org.apache.flink.kinesis.shaded.software.amazon.awssdk.http.apache.ApacheHttpClient.access > $500(ApacheHttpClient.java:104) ~[?:?] > at > > org.apache.flink.kinesis.shaded.software.amazon.awssdk.http.apache.ApacheHttpClient > $1.call(ApacheHttpClient.java:231) ~[?:?] > at > > org.apache.flink.kinesis.shaded.software.amazon.awssdk.http.apache.ApacheHttpClient > $1.call(ApacheHttpClient.java:228) ~[?:?] > at > > org.apache.flink.kinesis.shaded.software.amazon.awssdk.core.internal.util.MetricUtils.measureDurationUnsafe(MetricUtils.java: > 67) ~[?:?] > at > > org.apache.flink.kinesis.shaded.software.amazon.awssdk.core.internal.http.pipeline.stages.MakeHttpRequestStage.executeHttpRequest(MakeHttpRequestStage.java: > 77) ~[?:?] > at > > org.apache.flink.kinesis.shaded.software.amazon.awssdk.core.internal.http.pipeline.stages.MakeHttpRequestStage.execute(MakeHttpRequestStage.java: > 56) ~[?:?] > at > > org.apache.flink.kinesis.shaded.software.amazon.awssdk.core.internal.http.pipeline.stages.MakeHttpRequestStage.execute(MakeHttpRequestStage.java: > 39) ~[?:?] > at > > org.apache.flink.kinesis.shaded.software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder > $ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) > ~[?: > ?] > at > > org.apache.flink.kinesis.shaded.software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder > $ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) > ~[?: > ?] > at > > org.apache.flink.kinesis.shaded.software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder > $ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) > ~[?: > ?] > at > > org.apache.flink.kinesis.shaded.software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder > $ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) > ~[?: > ?] > at > > org.apache.flink.kinesis.shaded.software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java: > 72) ~[?:?] > at > > org.apache.flink.kinesis.shaded.software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java: > 42) ~[?:?] > at > > org.apache.flink.kinesis.shaded.software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java: > 78) ~[?:?] > at > > org.apache.flink.kinesis.shaded.software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java: > 40) ~[?:?] > at > > org.apache.flink.kinesis.shaded.software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptMetricCollectionStage.execute(ApiCallAttemptMetricCollectionStage.java: > 52) ~[?:?] > at > > org.apache.flink.kinesis.shaded.software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptMetricCollectionStage.execute(ApiCallAttemptMetricCollectionStage.java: > 37) ~[?:?] > at > > org.apache.flink.kinesis.shaded.software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java: > 81) ~[?:?] > at > > org.apache.flink.kinesis.shaded.software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java: > 36) ~[?:?] > at > > org.apache.flink.kinesis.shaded.software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder > $ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) > ~[?: > ?] > at > > org.apache.flink.kinesis.shaded.software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java: > 56) ~[?:?] > at > > org.apache.flink.kinesis.shaded.software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java: > 36) ~[?:?] > at > > org.apache.flink.kinesis.shaded.software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.executeWithTimer(ApiCallTimeoutTrackingStage.java: > 80) ~[?:?] > at > > org.apache.flink.kinesis.shaded.software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java: > 60) ~[?:?] > at > > org.apache.flink.kinesis.shaded.software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java: > 42) ~[?:?] > at > > org.apache.flink.kinesis.shaded.software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java: > 50) ~[?:?] > at > > org.apache.flink.kinesis.shaded.software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java: > 32) ~[?:?] > at > > org.apache.flink.kinesis.shaded.software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder > $ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) > ~[?: > ?] > at > > org.apache.flink.kinesis.shaded.software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder > $ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) > ~[?: > ?] > at > > org.apache.flink.kinesis.shaded.software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java: > 37) ~[?:?] > at > > org.apache.flink.kinesis.shaded.software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java: > 26) ~[?:?] > at > > org.apache.flink.kinesis.shaded.software.amazon.awssdk.core.internal.http.AmazonSyncHttpClient > $RequestExecutionBuilderImpl.execute(AmazonSyncHttpClient.java:198) ~[?:?] > at > > org.apache.flink.kinesis.shaded.software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.invoke(BaseSyncClientHandler.java: > 103) ~[?:?] > at > > org.apache.flink.kinesis.shaded.software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.doExecute(BaseSyncClientHandler.java: > 171) ~[?:?] > at > > org.apache.flink.kinesis.shaded.software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler. > lambda$execute$1(BaseSyncClientHandler.java:82) ~[?:?] > at > > org.apache.flink.kinesis.shaded.software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.measureApiCallSuccess(BaseSyncClientHandler.java: > 179) ~[?:?] > at > > org.apache.flink.kinesis.shaded.software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java: > 76) ~[?:?] > at > > org.apache.flink.kinesis.shaded.software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java: > 45) ~[?:?] > at > > org.apache.flink.kinesis.shaded.software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java: > 56) ~[?:?] > at > > org.apache.flink.kinesis.shaded.software.amazon.awssdk.services.sts.DefaultStsClient.assumeRoleWithWebIdentity(DefaultStsClient.java: > 757) ~[?:?] > at > > org.apache.flink.kinesis.shaded.software.amazon.awssdk.services.sts.auth.StsAssumeRoleWithWebIdentityCredentialsProvider.getUpdatedCredentials(StsAssumeRoleWithWebIdentityCredentialsProvider.java: > 74) ~[?:?] > at > > org.apache.flink.kinesis.shaded.software.amazon.awssdk.services.sts.auth.StsCredentialsProvider.updateSessionCredentials(StsCredentialsProvider.java: > 92) ~[?:?] > at > > org.apache.flink.kinesis.shaded.software.amazon.awssdk.utils.cache.CachedSupplier. > lambda$jitteredPrefetchValueSupplier$8(CachedSupplier.java:300) ~[?:?] > at > > org.apache.flink.kinesis.shaded.software.amazon.awssdk.utils.cache.CachedSupplier > $PrefetchStrategy.fetch(CachedSupplier.java:448) ~[?:?] > at > > org.apache.flink.kinesis.shaded.software.amazon.awssdk.utils.cache.CachedSupplier.refreshCache(CachedSupplier.java: > 208) ~[?:?] > at > > org.apache.flink.kinesis.shaded.software.amazon.awssdk.utils.cache.CachedSupplier.get(CachedSupplier.java: > 135) ~[?:?] > at > > org.apache.flink.kinesis.shaded.software.amazon.awssdk.services.sts.auth.StsCredentialsProvider.resolveCredentials(StsCredentialsProvider.java: > 105) ~[?:?] > at > > org.apache.flink.kinesis.shaded.software.amazon.awssdk.services.sts.internal.StsWebIdentityCredentialsProviderFactory > $ > > StsWebIdentityCredentialsProvider.resolveCredentials(StsWebIdentityCredentialsProviderFactory.java: > 109) ~[?:?] > at > > org.apache.flink.kinesis.shaded.software.amazon.awssdk.auth.credentials.WebIdentityTokenFileCredentialsProvider.resolveCredentials(WebIdentityTokenFileCredentialsProvider.java: > 143) ~[?:?] > at > > org.apache.flink.kinesis.shaded.software.amazon.awssdk.auth.credentials.AwsCredentialsProviderChain.resolveCredentials(AwsCredentialsProviderChain.java: > 90) ~[?:?] > at > > org.apache.flink.kinesis.shaded.software.amazon.awssdk.auth.credentials.internal.LazyAwsCredentialsProvider.resolveCredentials(LazyAwsCredentialsProvider.java: > 45) ~[?:?] > at > > org.apache.flink.kinesis.shaded.software.amazon.awssdk.auth.credentials.DefaultCredentialsProvider.resolveCredentials(DefaultCredentialsProvider.java: > 128) ~[?:?] > at > > org.apache.flink.kinesis.shaded.software.amazon.awssdk.core.internal.util.MetricUtils.measureDuration(MetricUtils.java: > 54) ~[?:?] > at > > org.apache.flink.kinesis.shaded.software.amazon.awssdk.awscore.internal.authcontext.AwsCredentialsAuthorizationStrategy.resolveCredentials(AwsCredentialsAuthorizationStrategy.java: > 100) ~[?:?] > at > > org.apache.flink.kinesis.shaded.software.amazon.awssdk.awscore.internal.authcontext.AwsCredentialsAuthorizationStrategy.addCredentialsToExecutionAttributes(AwsCredentialsAuthorizationStrategy.java: > 77) ~[?:?] > at > > org.apache.flink.kinesis.shaded.software.amazon.awssdk.awscore.internal.AwsExecutionContextBuilder.invokeInterceptorsAndCreateExecutionContext(AwsExecutionContextBuilder.java: > 125) ~[?:?] > at > > org.apache.flink.kinesis.shaded.software.amazon.awssdk.awscore.client.handler.AwsAsyncClientHandler.invokeInterceptorsAndCreateExecutionContext(AwsAsyncClientHandler.java: > 65) ~[?:?] > at > > org.apache.flink.kinesis.shaded.software.amazon.awssdk.core.internal.handler.BaseAsyncClientHandler. > lambda$execute$3(BaseAsyncClientHandler.java:118) ~[?:?] > at > > org.apache.flink.kinesis.shaded.software.amazon.awssdk.core.internal.handler.BaseAsyncClientHandler.measureApiCallSuccess(BaseAsyncClientHandler.java: > 291) ~[?:?] > at > > org.apache.flink.kinesis.shaded.software.amazon.awssdk.core.internal.handler.BaseAsyncClientHandler.execute(BaseAsyncClientHandler.java: > 91) ~[?:?] > at > > org.apache.flink.kinesis.shaded.software.amazon.awssdk.awscore.client.handler.AwsAsyncClientHandler.execute(AwsAsyncClientHandler.java: > 59) ~[?:?] > at > > org.apache.flink.kinesis.shaded.software.amazon.awssdk.services.kinesis.DefaultKinesisAsyncClient.subscribeToShard(DefaultKinesisAsyncClient.java: > 2730) ~[?:?] > at > > org.apache.flink.streaming.connectors.kinesis.proxy.KinesisProxyAsyncV2.subscribeToShard(KinesisProxyAsyncV2.java: > 66) ~[?:?] > at > > org.apache.flink.streaming.connectors.kinesis.internals.publisher.fanout.FanOutShardSubscriber.openSubscriptionToShard(FanOutShardSubscriber.java: > 250) ~[?:?] > at > > org.apache.flink.streaming.connectors.kinesis.internals.publisher.fanout.FanOutShardSubscriber.subscribeToShardAndConsumeRecords(FanOutShardSubscriber.java: > 196) ~[?:?] > at > > org.apache.flink.streaming.connectors.kinesis.internals.publisher.fanout.FanOutRecordPublisher.runWithBackoff(FanOutRecordPublisher.java: > 176) ~[?:?] > ... 7 more > > On Wed, Feb 28, 2024 at 6:21 PM Aleksandr Pilipenko <z3d...@gmail.com> > wrote: > > > Hi, > > > > Could you please provide more information on the error you are observing? > > Attached file does not have anything related to Kinesis or any errors. > > > > Best, > > Aleksandr > > > > On Wed, 28 Feb 2024 at 02:28, Xiaolong Wang > > <xiaolong.w...@smartnews.com.invalid> wrote: > > > > > Hi, > > > > > > I used the flink-connector-kinesis (4.0.2-1.18) to consume from > Kinesis. > > > The job can start but will fail within 1 hour. Detailed error log > > > is attached. > > > > > > When I changed the version of the flink-connector-kinesis to `1.15.2` , > > > everything settled. > > > > > > Any idea to fix it ? > > > > > > > > >
