I think I found the root cause https://bugs.alpinelinux.org/issues/10126
I have to re-install nss after apk update/upgrade Hao Sun On Sun, Nov 11, 2018 at 10:50 AM Ufuk Celebi <u...@apache.org> wrote: > Hey Hao, > > 1) Regarding Hadoop S3: are you using the repackaged Hadoop S3 > dependency from the /opt folder of the Flink distribution? Or the > actual Hadoop implementation? If latter, would you mind also running > it with the one that comes packaged with Flink? For this you can > remove all Hadoop-related configuration in your flink-conf.yaml and > copy the Hadoop S3 dependency from /opt to /lib and configure it [1]. > > 2) Could you please share your complete Flink configuration for when > you tried to run with Presto S3? If you don't want to share this > publicly, feel free to share it privately with me. I'm curious to see > whether we can reproduce this. > > – Ufuk > > [1] > https://ci.apache.org/projects/flink/flink-docs-release-1.6/ops/deployment/aws.html#shaded-hadooppresto-s3-file-systems-recommended > <https://ci.apache.org/projects/flink/flink-docs-release-1.6/ops/deployment/aws.html#shaded-hadooppresto-s3-file-systems-recommended> > On Sat, Nov 10, 2018 at 4:07 PM Hao Sun <ha...@zendesk.com> wrote: > > > > Hi Ufuk, thanks for checking. I am using openJDK 1.8_171, I still have > the same issue with presto. > > > > - why checkpoint is not starting from 1? old chk stored in ZK caused it, > I cleaned it up, but not very helpful > > - I switched to Flink + Hadoop28, and used hadoop s3, with no other > changes, check pointing is working with the hadoop flavour. > > > > On Fri, Nov 9, 2018 at 2:02 PM Ufuk Celebi <u...@apache.org> wrote: > >> > >> Hey Hao Sun, > >> > >> - Is this an intermittent failure or permanent? The logs indicate that > >> some checkpoints completed before the error occurs (e.g. checkpoint > >> numbers are greater than 1). > >> > >> - Which Java versions are you using? And which Java image? I've > >> Googled similar issues that seem to be related to the JVM, e.g. [1]. > >> > >> Best, > >> > >> Ufuk > >> > >> [1] > https://dev.lucee.org/t/could-not-initialize-class-sun-security-ssl-sslcontextimp/3972 > <https://dev.lucee.org/t/could-not-initialize-class-sun-security-ssl-sslcontextimp/3972> > >> > >> > >> On Thu, Nov 8, 2018 at 8:55 PM Hao Sun <ha...@zendesk.com> wrote: > >> > > >> > Thanks, any insight/help here is appreciated. > >> > > >> > On Thu, Nov 8, 2018 at 4:38 AM Dawid Wysakowicz < > dwysakow...@apache.org> wrote: > >> >> > >> >> Hi Hao, > >> >> > >> >> I am not sure, what might be wrong, but I've cc'ed Gary and Kostas > who were recently working with S3, maybe they will have some ideas. > >> >> > >> >> Best, > >> >> > >> >> Dawid > >> >> > >> >> On 03/11/2018 03:09, Hao Sun wrote: > >> >> > >> >> Same environment, new error. > >> >> > >> >> I can run the same docker image with my local Mac, but on K8S, this > gives me this error. > >> >> I can not think of any difference between local Docker and K8S > Docker. > >> >> > >> >> Any hint will be helpful. Thanks > >> >> > >> >> ============ > >> >> > >> >> 2018-11-02 23:29:32,981 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Job > ConnectedStreams maxwell.accounts (00000000000000000000000000000000) > switched from state RUNNING to FAILING. > >> >> AsynchronousException{java.lang.Exception: Could not materialize > checkpoint 235 for operator Source: KafkaSource(maxwell.accounts) -> > MaxwellFilter->Maxwell(maxwell.accounts) -> > FixedDelayWatermark(maxwell.accounts) -> > MaxwellFPSEvent->InfluxDBData(maxwell.accounts) -> Sink: > influxdbSink(maxwell.accounts) (1/1).} > >> >> at > org.apache.flink.streaming.runtime.tasks.StreamTask$AsyncCheckpointExceptionHandler.tryHandleCheckpointException(StreamTask.java:1153) > >> >> at > org.apache.flink.streaming.runtime.tasks.StreamTask$AsyncCheckpointRunnable.handleExecutionException(StreamTask.java:947) > >> >> at > org.apache.flink.streaming.runtime.tasks.StreamTask$AsyncCheckpointRunnable.run(StreamTask.java:884) > >> >> at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > >> >> at java.util.concurrent.FutureTask.run(FutureTask.java:266) > >> >> at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > >> >> at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > >> >> at java.lang.Thread.run(Thread.java:748) > >> >> Caused by: java.lang.Exception: Could not materialize checkpoint 235 > for operator Source: KafkaSource(maxwell.accounts) -> > MaxwellFilter->Maxwell(maxwell.accounts) -> > FixedDelayWatermark(maxwell.accounts) -> > MaxwellFPSEvent->InfluxDBData(maxwell.accounts) -> Sink: > influxdbSink(maxwell.accounts) (1/1). > >> >> at > org.apache.flink.streaming.runtime.tasks.StreamTask$AsyncCheckpointRunnable.handleExecutionException(StreamTask.java:942) > >> >> ... 6 more > >> >> Caused by: java.util.concurrent.ExecutionException: > java.lang.NoClassDefFoundError: Could not initialize class > sun.security.ssl.SSLSessionImpl > >> >> at java.util.concurrent.FutureTask.report(FutureTask.java:122) > >> >> at java.util.concurrent.FutureTask.get(FutureTask.java:192) > >> >> at > org.apache.flink.util.FutureUtil.runIfNotDoneAndGet(FutureUtil.java:53) > >> >> > >> >> at > org.apache.flink.streaming.api.operators.OperatorSnapshotFinalizer.<init>( > http://OperatorSnapshotFinalizer.java:53 > <http://OperatorSnapshotFinalizer.java:53>) > >> >> > >> >> at > org.apache.flink.streaming.runtime.tasks.StreamTask$AsyncCheckpointRunnable.run(StreamTask.java:853) > >> >> ... 5 more > >> >> Caused by: java.lang.NoClassDefFoundError: Could not initialize > class sun.security.ssl.SSLSessionImpl > >> >> at sun.security.ssl.SSLSocketImpl.init(SSLSocketImpl.java:604) > >> >> > >> >> at sun.security.ssl.SSLSocketImpl.<init>( > http://SSLSocketImpl.java:572 > <http://SSLSocketImpl.java:572>) > >> >> > >> >> at > sun.security.ssl.SSLSocketFactoryImpl.createSocket(SSLSocketFactoryImpl.java:110) > >> >> at > org.apache.flink.fs.s3presto.shaded.org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:365) > >> >> at > org.apache.flink.fs.s3presto.shaded.org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:355) > >> >> at > org.apache.flink.fs.s3presto.shaded.com.amazonaws.http.conn.ssl.SdkTLSSocketFactory.connectSocket(SdkTLSSocketFactory.java:132) > >> >> at > org.apache.flink.fs.s3presto.shaded.org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) > >> >> at > org.apache.flink.fs.s3presto.shaded.org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:359) > >> >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > >> >> at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > >> >> at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > >> >> at java.lang.reflect.Method.invoke(Method.java:498) > >> >> at > org.apache.flink.fs.s3presto.shaded.com.amazonaws.http.conn.ClientConnectionManagerFactory$Handler.invoke(ClientConnectionManagerFactory.java:76) > >> >> at > org.apache.flink.fs.s3presto.shaded.com.amazonaws.http.conn.$Proxy4.connect(Unknown > Source) > >> >> at > org.apache.flink.fs.s3presto.shaded.org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381) > >> >> at > org.apache.flink.fs.s3presto.shaded.org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237) > >> >> at > org.apache.flink.fs.s3presto.shaded.org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) > >> >> at > org.apache.flink.fs.s3presto.shaded.org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) > >> >> at > org.apache.flink.fs.s3presto.shaded.org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) > >> >> at > org.apache.flink.fs.s3presto.shaded.org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56) > >> >> at > org.apache.flink.fs.s3presto.shaded.com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72) > >> >> at > org.apache.flink.fs.s3presto.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1189) > >> >> at > org.apache.flink.fs.s3presto.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1029) > >> >> at > org.apache.flink.fs.s3presto.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:741) > >> >> at > org.apache.flink.fs.s3presto.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:715) > >> >> at > org.apache.flink.fs.s3presto.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:697) > >> >> at > org.apache.flink.fs.s3presto.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:665) > >> >> at > org.apache.flink.fs.s3presto.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:647) > >> >> at > org.apache.flink.fs.s3presto.shaded.com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:511) > >> >> at > org.apache.flink.fs.s3presto.shaded.com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4227) > >> >> at > org.apache.flink.fs.s3presto.shaded.com.amazonaws.services.s3.AmazonS3Client.getBucketRegionViaHeadRequest(AmazonS3Client.java:4988) > >> >> at > org.apache.flink.fs.s3presto.shaded.com.amazonaws.services.s3.AmazonS3Client.fetchRegionFromCache(AmazonS3Client.java:4962) > >> >> at > org.apache.flink.fs.s3presto.shaded.com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4211) > >> >> at > org.apache.flink.fs.s3presto.shaded.com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4174) > >> >> at > org.apache.flink.fs.s3presto.shaded.com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:1253) > >> >> at > org.apache.flink.fs.s3presto.shaded.com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:1228) > >> >> at > org.apache.flink.fs.s3presto.shaded.com.facebook.presto.hive.PrestoS3FileSystem.lambda$getS3ObjectMetadata$2(PrestoS3FileSystem.java:559) > >> >> at > org.apache.flink.fs.s3presto.shaded.com.facebook.presto.hive.RetryDriver.run(RetryDriver.java:138) > >> >> at > org.apache.flink.fs.s3presto.shaded.com.facebook.presto.hive.PrestoS3FileSystem.getS3ObjectMetadata(PrestoS3FileSystem.java:556) > >> >> at > org.apache.flink.fs.s3presto.shaded.com.facebook.presto.hive.PrestoS3FileSystem.getFileStatus(PrestoS3FileSystem.java:307) > >> >> at > org.apache.flink.fs.s3presto.shaded.org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1426) > >> >> at > org.apache.flink.fs.s3presto.shaded.com.facebook.presto.hive.PrestoS3FileSystem.create(PrestoS3FileSystem.java:352) > >> >> at > org.apache.flink.fs.s3presto.shaded.org.apache.hadoop.fs.FileSystem.create(FileSystem.java:911) > >> >> at > org.apache.flink.fs.s3presto.shaded.org.apache.hadoop.fs.FileSystem.create(FileSystem.java:892) > >> >> at > org.apache.flink.fs.s3presto.shaded.org.apache.hadoop.fs.FileSystem.create(FileSystem.java:789) > >> >> at > org.apache.flink.fs.s3presto.shaded.org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.create(HadoopFileSystem.java:141) > >> >> at > org.apache.flink.fs.s3presto.shaded.org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.create(HadoopFileSystem.java:37) > >> >> at > org.apache.flink.core.fs.SafetyNetWrapperFileSystem.create(SafetyNetWrapperFileSystem.java:126) > >> >> at > org.apache.flink.core.fs.EntropyInjector.createEntropyAware(EntropyInjector.java:61) > >> >> at > org.apache.flink.runtime.state.filesystem.FsCheckpointStreamFactory$FsCheckpointStateOutputStream.createStream(FsCheckpointStreamFactory.java:350) > >> >> at > org.apache.flink.runtime.state.filesystem.FsCheckpointStreamFactory$FsCheckpointStateOutputStream.flush(FsCheckpointStreamFactory.java:228) > >> >> at > org.apache.flink.runtime.state.filesystem.FsCheckpointStreamFactory$FsCheckpointStateOutputStream.write(FsCheckpointStreamFactory.java:212) > >> >> at java.io.DataOutputStream.write(DataOutputStream.java:107) > >> >> at > org.apache.flink.api.common.typeutils.TypeSerializerSerializationUtil.writeSerializersAndConfigsWithResilience(TypeSerializerSerializationUtil.java:161) > >> >> at > org.apache.flink.runtime.state.metainfo.StateMetaInfoSnapshotReadersWriters$CurrentWriterImpl.writeStateMetaInfoSnapshot(StateMetaInfoSnapshotReadersWriters.java:182) > >> >> at > org.apache.flink.runtime.state.OperatorBackendSerializationProxy.writeStateMetaInfoSnapshots(OperatorBackendSerializationProxy.java:85) > >> >> at > org.apache.flink.runtime.state.OperatorBackendSerializationProxy.write(OperatorBackendSerializationProxy.java:76) > >> >> at > org.apache.flink.runtime.state.DefaultOperatorStateBackend$1.performOperation(DefaultOperatorStateBackend.java:418) > >> >> at > org.apache.flink.runtime.state.DefaultOperatorStateBackend$1.performOperation(DefaultOperatorStateBackend.java:359) > >> >> at > org.apache.flink.runtime.io.async.AbstractAsyncCallableWithResources.call(AbstractAsyncCallableWithResources.java:75) > >> >> at java.util.concurrent.FutureTask.run(FutureTask.java:266) > >> >> at > org.apache.flink.util.FutureUtil.runIfNotDoneAndGet(FutureUtil.java:50) > >> >> ... 7 more > >> >> 2018-11-02 23:29:32,983 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Try to restart or > fail the job ConnectedStreams maxwell.accounts > (00000000000000000000000000000000) if no longer possible. > >> >> > >> >> ========================= > >> >> > >> >> On Thu, Nov 1, 2018 at 9:22 PM Hao Sun <ha...@zendesk.com> wrote: > >> >>> > >> >>> I am on Flink 1.6.2 (no Hadoop, in docker + K8S), using rocksdb and > S3 (presto) > >> >>> I got this error when flink creating a checking point > >> >>> > >> >>> > >> >>> =========== > >> >>> 2018-11-02 04:00:55,011 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Job > ConnectedStreams maxwell.accounts (00000000000000000000000000000000) > switched from state RUNNING to FAILING. > >> >>> AsynchronousException{java.lang.Exception: Could not materialize > checkpoint 130 for operator Source: KafkaSource(maxwell.accounts) -> > MaxwellFilter->Maxwell(maxwell.accounts) -> > FixedDelayWatermark(maxwell.accounts) -> > MaxwellFPSEvent->InfluxDBData(maxwell.accounts) -> Sink: > influxdbSink(maxwell.accounts) (1/1).} > >> >>> at > org.apache.flink.streaming.runtime.tasks.StreamTask$AsyncCheckpointExceptionHandler.tryHandleCheckpointException(StreamTask.java:1153) > >> >>> at > org.apache.flink.streaming.runtime.tasks.StreamTask$AsyncCheckpointRunnable.handleExecutionException(StreamTask.java:947) > >> >>> at > org.apache.flink.streaming.runtime.tasks.StreamTask$AsyncCheckpointRunnable.run(StreamTask.java:884) > >> >>> at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > >> >>> at java.util.concurrent.FutureTask.run(FutureTask.java:266) > >> >>> at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > >> >>> at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > >> >>> at java.lang.Thread.run(Thread.java:748) > >> >>> Caused by: java.lang.Exception: Could not materialize checkpoint > 130 for operator Source: KafkaSource(maxwell.accounts) -> > MaxwellFilter->Maxwell(maxwell.accounts) -> > FixedDelayWatermark(maxwell.accounts) -> > MaxwellFPSEvent->InfluxDBData(maxwell.accounts) -> Sink: > influxdbSink(maxwell.accounts) (1/1). > >> >>> at > org.apache.flink.streaming.runtime.tasks.StreamTask$AsyncCheckpointRunnable.handleExecutionException(StreamTask.java:942) > >> >>> ... 6 more > >> >>> Caused by: java.util.concurrent.ExecutionException: > java.lang.ExceptionInInitializerError > >> >>> at java.util.concurrent.FutureTask.report(FutureTask.java:122) > >> >>> at java.util.concurrent.FutureTask.get(FutureTask.java:192) > >> >>> at > org.apache.flink.util.FutureUtil.runIfNotDoneAndGet(FutureUtil.java:53) > >> >>> > >> >>> at > org.apache.flink.streaming.api.operators.OperatorSnapshotFinalizer.<init>( > http://OperatorSnapshotFinalizer.java:53 > <http://OperatorSnapshotFinalizer.java:53>) > >> >>> > >> >>> > >> >>> at > org.apache.flink.streaming.runtime.tasks.StreamTask$AsyncCheckpointRunnable.run(StreamTask.java:853) > >> >>> ... 5 more > >> >>> Caused by: java.lang.ExceptionInInitializerError > >> >>> > >> >>> at sun.security.ssl.SSLSessionImpl.<init>( > http://SSLSessionImpl.java:183 > <http://SSLSessionImpl.java:183>) > >> >>> at sun.security.ssl.SSLSessionImpl.<init>( > http://SSLSessionImpl.java:148 > <http://SSLSessionImpl.java:148>) > >> >>> at sun.security.ssl.SSLSessionImpl.<clinit>( > http://SSLSessionImpl.java:79 > <http://SSLSessionImpl.java:79>) > >> >>> at sun.security.ssl.SSLSocketImpl.init(SSLSocketImpl.java:604) > >> >>> at sun.security.ssl.SSLSocketImpl.<init>( > http://SSLSocketImpl.java:572 > <http://SSLSocketImpl.java:572>) > >> >>> > >> >>> > >> >>> at > sun.security.ssl.SSLSocketFactoryImpl.createSocket(SSLSocketFactoryImpl.java:110) > >> >>> at > org.apache.flink.fs.s3presto.shaded.org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:365) > >> >>> at > org.apache.flink.fs.s3presto.shaded.org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:355) > >> >>> at > org.apache.flink.fs.s3presto.shaded.com.amazonaws.http.conn.ssl.SdkTLSSocketFactory.connectSocket(SdkTLSSocketFactory.java:132) > >> >>> at > org.apache.flink.fs.s3presto.shaded.org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) > >> >>> at > org.apache.flink.fs.s3presto.shaded.org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:359) > >> >>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > >> >>> at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > >> >>> at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > >> >>> at java.lang.reflect.Method.invoke(Method.java:498) > >> >>> at > org.apache.flink.fs.s3presto.shaded.com.amazonaws.http.conn.ClientConnectionManagerFactory$Handler.invoke(ClientConnectionManagerFactory.java:76) > >> >>> at > org.apache.flink.fs.s3presto.shaded.com.amazonaws.http.conn.$Proxy4.connect(Unknown > Source) > >> >>> at > org.apache.flink.fs.s3presto.shaded.org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381) > >> >>> at > org.apache.flink.fs.s3presto.shaded.org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237) > >> >>> at > org.apache.flink.fs.s3presto.shaded.org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) > >> >>> at > org.apache.flink.fs.s3presto.shaded.org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) > >> >>> at > org.apache.flink.fs.s3presto.shaded.org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) > >> >>> at > org.apache.flink.fs.s3presto.shaded.org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56) > >> >>> at > org.apache.flink.fs.s3presto.shaded.com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72) > >> >>> at > org.apache.flink.fs.s3presto.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1189) > >> >>> at > org.apache.flink.fs.s3presto.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1029) > >> >>> at > org.apache.flink.fs.s3presto.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:741) > >> >>> at > org.apache.flink.fs.s3presto.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:715) > >> >>> at > org.apache.flink.fs.s3presto.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:697) > >> >>> at > org.apache.flink.fs.s3presto.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:665) > >> >>> at > org.apache.flink.fs.s3presto.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:647) > >> >>> at > org.apache.flink.fs.s3presto.shaded.com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:511) > >> >>> at > org.apache.flink.fs.s3presto.shaded.com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4227) > >> >>> at > org.apache.flink.fs.s3presto.shaded.com.amazonaws.services.s3.AmazonS3Client.getBucketRegionViaHeadRequest(AmazonS3Client.java:4988) > >> >>> at > org.apache.flink.fs.s3presto.shaded.com.amazonaws.services.s3.AmazonS3Client.fetchRegionFromCache(AmazonS3Client.java:4962) > >> >>> at > org.apache.flink.fs.s3presto.shaded.com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4211) > >> >>> at > org.apache.flink.fs.s3presto.shaded.com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4174) > >> >>> at > org.apache.flink.fs.s3presto.shaded.com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:1253) > >> >>> at > org.apache.flink.fs.s3presto.shaded.com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:1228) > >> >>> at > org.apache.flink.fs.s3presto.shaded.com.facebook.presto.hive.PrestoS3FileSystem.lambda$getS3ObjectMetadata$2(PrestoS3FileSystem.java:559) > >> >>> at > org.apache.flink.fs.s3presto.shaded.com.facebook.presto.hive.RetryDriver.run(RetryDriver.java:138) > >> >>> at > org.apache.flink.fs.s3presto.shaded.com.facebook.presto.hive.PrestoS3FileSystem.getS3ObjectMetadata(PrestoS3FileSystem.java:556) > >> >>> at > org.apache.flink.fs.s3presto.shaded.com.facebook.presto.hive.PrestoS3FileSystem.getFileStatus(PrestoS3FileSystem.java:307) > >> >>> at > org.apache.flink.fs.s3presto.shaded.org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1426) > >> >>> at > org.apache.flink.fs.s3presto.shaded.com.facebook.presto.hive.PrestoS3FileSystem.create(PrestoS3FileSystem.java:352) > >> >>> at > org.apache.flink.fs.s3presto.shaded.org.apache.hadoop.fs.FileSystem.create(FileSystem.java:911) > >> >>> at > org.apache.flink.fs.s3presto.shaded.org.apache.hadoop.fs.FileSystem.create(FileSystem.java:892) > >> >>> at > org.apache.flink.fs.s3presto.shaded.org.apache.hadoop.fs.FileSystem.create(FileSystem.java:789) > >> >>> at > org.apache.flink.fs.s3presto.shaded.org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.create(HadoopFileSystem.java:141) > >> >>> at > org.apache.flink.fs.s3presto.shaded.org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.create(HadoopFileSystem.java:37) > >> >>> at > org.apache.flink.core.fs.SafetyNetWrapperFileSystem.create(SafetyNetWrapperFileSystem.java:126) > >> >>> at > org.apache.flink.core.fs.EntropyInjector.createEntropyAware(EntropyInjector.java:61) > >> >>> at > org.apache.flink.runtime.state.filesystem.FsCheckpointStreamFactory$FsCheckpointStateOutputStream.createStream(FsCheckpointStreamFactory.java:350) > >> >>> at > org.apache.flink.runtime.state.filesystem.FsCheckpointStreamFactory$FsCheckpointStateOutputStream.flush(FsCheckpointStreamFactory.java:228) > >> >>> at > org.apache.flink.runtime.state.filesystem.FsCheckpointStreamFactory$FsCheckpointStateOutputStream.write(FsCheckpointStreamFactory.java:212) > >> >>> at java.io.DataOutputStream.write(DataOutputStream.java:107) > >> >>> at > org.apache.flink.api.common.typeutils.TypeSerializerSerializationUtil.writeSerializersAndConfigsWithResilience(TypeSerializerSerializationUtil.java:161) > >> >>> at > org.apache.flink.runtime.state.metainfo.StateMetaInfoSnapshotReadersWriters$CurrentWriterImpl.writeStateMetaInfoSnapshot(StateMetaInfoSnapshotReadersWriters.java:182) > >> >>> at > org.apache.flink.runtime.state.OperatorBackendSerializationProxy.writeStateMetaInfoSnapshots(OperatorBackendSerializationProxy.java:85) > >> >>> at > org.apache.flink.runtime.state.OperatorBackendSerializationProxy.write(OperatorBackendSerializationProxy.java:76) > >> >>> at > org.apache.flink.runtime.state.DefaultOperatorStateBackend$1.performOperation(DefaultOperatorStateBackend.java:418) > >> >>> at > org.apache.flink.runtime.state.DefaultOperatorStateBackend$1.performOperation(DefaultOperatorStateBackend.java:359) > >> >>> at > org.apache.flink.runtime.io.async.AbstractAsyncCallableWithResources.call(AbstractAsyncCallableWithResources.java:75) > >> >>> at java.util.concurrent.FutureTask.run(FutureTask.java:266) > >> >>> at > org.apache.flink.util.FutureUtil.runIfNotDoneAndGet(FutureUtil.java:50) > >> >>> ... 7 more > >> >>> Caused by: java.security.ProviderException: Could not initialize NSS > >> >>> > >> >>> at sun.security.pkcs11.SunPKCS11.<init>(http://SunPKCS11.java:223 > <http://SunPKCS11.java:223>) > >> >>> at sun.security.pkcs11.SunPKCS11.<init>(http://SunPKCS11.java:103 > <http://SunPKCS11.java:103>) > >> >>> > >> >>> > >> >>> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native > Method) > >> >>> at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > >> >>> at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > >> >>> at java.lang.reflect.Constructor.newInstance(Constructor.java:423) > >> >>> at sun.security.jca.ProviderConfig$2.run(ProviderConfig.java:224) > >> >>> at sun.security.jca.ProviderConfig$2.run(ProviderConfig.java:206) > >> >>> at java.security.AccessController.doPrivileged(Native Method) > >> >>> at > sun.security.jca.ProviderConfig.doLoadProvider(ProviderConfig.java:206) > >> >>> at > sun.security.jca.ProviderConfig.getProvider(ProviderConfig.java:187) > >> >>> at sun.security.jca.ProviderList.getProvider(ProviderList.java:233) > >> >>> at sun.security.jca.ProviderList.getIndex(ProviderList.java:263) > >> >>> at > sun.security.jca.ProviderList.getProviderConfig(ProviderList.java:247) > >> >>> at sun.security.jca.ProviderList.getProvider(ProviderList.java:253) > >> >>> at java.security.Security.getProvider(Security.java:501) > >> >>> > >> >>> at sun.security.ssl.SignatureAndHashAlgorithm.<clinit>( > http://SignatureAndHashAlgorithm.java:415 > <http://SignatureAndHashAlgorithm.java:415>) > >> >>> > >> >>> > >> >>> ... 72 more > >> >>> Caused by: java.io.IOException: NSS is already initialized > >> >>> at sun.security.pkcs11.Secmod.initialize(Secmod.java:173) > >> >>> > >> >>> at sun.security.pkcs11.SunPKCS11.<init>(http://SunPKCS11.java:218 > <http://SunPKCS11.java:218>) > >> >>> ... 88 more > >> >>> > >> >>> =========== >
