Hey Hao, 1) Regarding Hadoop S3: are you using the repackaged Hadoop S3 dependency from the /opt folder of the Flink distribution? Or the actual Hadoop implementation? If latter, would you mind also running it with the one that comes packaged with Flink? For this you can remove all Hadoop-related configuration in your flink-conf.yaml and copy the Hadoop S3 dependency from /opt to /lib and configure it [1].
2) Could you please share your complete Flink configuration for when you tried to run with Presto S3? If you don't want to share this publicly, feel free to share it privately with me. I'm curious to see whether we can reproduce this. – Ufuk [1] https://ci.apache.org/projects/flink/flink-docs-release-1.6/ops/deployment/aws.html#shaded-hadooppresto-s3-file-systems-recommended On Sat, Nov 10, 2018 at 4:07 PM Hao Sun <ha...@zendesk.com> wrote: > > Hi Ufuk, thanks for checking. I am using openJDK 1.8_171, I still have the > same issue with presto. > > - why checkpoint is not starting from 1? old chk stored in ZK caused it, I > cleaned it up, but not very helpful > - I switched to Flink + Hadoop28, and used hadoop s3, with no other changes, > check pointing is working with the hadoop flavour. > > On Fri, Nov 9, 2018 at 2:02 PM Ufuk Celebi <u...@apache.org> wrote: >> >> Hey Hao Sun, >> >> - Is this an intermittent failure or permanent? The logs indicate that >> some checkpoints completed before the error occurs (e.g. checkpoint >> numbers are greater than 1). >> >> - Which Java versions are you using? And which Java image? I've >> Googled similar issues that seem to be related to the JVM, e.g. [1]. >> >> Best, >> >> Ufuk >> >> [1] >> https://dev.lucee.org/t/could-not-initialize-class-sun-security-ssl-sslcontextimp/3972 >> >> >> On Thu, Nov 8, 2018 at 8:55 PM Hao Sun <ha...@zendesk.com> wrote: >> > >> > Thanks, any insight/help here is appreciated. >> > >> > On Thu, Nov 8, 2018 at 4:38 AM Dawid Wysakowicz <dwysakow...@apache.org> >> > wrote: >> >> >> >> Hi Hao, >> >> >> >> I am not sure, what might be wrong, but I've cc'ed Gary and Kostas who >> >> were recently working with S3, maybe they will have some ideas. >> >> >> >> Best, >> >> >> >> Dawid >> >> >> >> On 03/11/2018 03:09, Hao Sun wrote: >> >> >> >> Same environment, new error. >> >> >> >> I can run the same docker image with my local Mac, but on K8S, this gives >> >> me this error. >> >> I can not think of any difference between local Docker and K8S Docker. >> >> >> >> Any hint will be helpful. Thanks >> >> >> >> ============ >> >> >> >> 2018-11-02 23:29:32,981 INFO >> >> org.apache.flink.runtime.executiongraph.ExecutionGraph - Job >> >> ConnectedStreams maxwell.accounts (00000000000000000000000000000000) >> >> switched from state RUNNING to FAILING. >> >> AsynchronousException{java.lang.Exception: Could not materialize >> >> checkpoint 235 for operator Source: KafkaSource(maxwell.accounts) -> >> >> MaxwellFilter->Maxwell(maxwell.accounts) -> >> >> FixedDelayWatermark(maxwell.accounts) -> >> >> MaxwellFPSEvent->InfluxDBData(maxwell.accounts) -> Sink: >> >> influxdbSink(maxwell.accounts) (1/1).} >> >> at >> >> org.apache.flink.streaming.runtime.tasks.StreamTask$AsyncCheckpointExceptionHandler.tryHandleCheckpointException(StreamTask.java:1153) >> >> at >> >> org.apache.flink.streaming.runtime.tasks.StreamTask$AsyncCheckpointRunnable.handleExecutionException(StreamTask.java:947) >> >> at >> >> org.apache.flink.streaming.runtime.tasks.StreamTask$AsyncCheckpointRunnable.run(StreamTask.java:884) >> >> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) >> >> at java.util.concurrent.FutureTask.run(FutureTask.java:266) >> >> at >> >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) >> >> at >> >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) >> >> at java.lang.Thread.run(Thread.java:748) >> >> Caused by: java.lang.Exception: Could not materialize checkpoint 235 for >> >> operator Source: KafkaSource(maxwell.accounts) -> >> >> MaxwellFilter->Maxwell(maxwell.accounts) -> >> >> FixedDelayWatermark(maxwell.accounts) -> >> >> MaxwellFPSEvent->InfluxDBData(maxwell.accounts) -> Sink: >> >> influxdbSink(maxwell.accounts) (1/1). >> >> at >> >> org.apache.flink.streaming.runtime.tasks.StreamTask$AsyncCheckpointRunnable.handleExecutionException(StreamTask.java:942) >> >> ... 6 more >> >> Caused by: java.util.concurrent.ExecutionException: >> >> java.lang.NoClassDefFoundError: Could not initialize class >> >> sun.security.ssl.SSLSessionImpl >> >> at java.util.concurrent.FutureTask.report(FutureTask.java:122) >> >> at java.util.concurrent.FutureTask.get(FutureTask.java:192) >> >> at org.apache.flink.util.FutureUtil.runIfNotDoneAndGet(FutureUtil.java:53) >> >> >> >> at >> >> org.apache.flink.streaming.api.operators.OperatorSnapshotFinalizer.<init>(http://OperatorSnapshotFinalizer.java:53) >> >> >> >> at >> >> org.apache.flink.streaming.runtime.tasks.StreamTask$AsyncCheckpointRunnable.run(StreamTask.java:853) >> >> ... 5 more >> >> Caused by: java.lang.NoClassDefFoundError: Could not initialize class >> >> sun.security.ssl.SSLSessionImpl >> >> at sun.security.ssl.SSLSocketImpl.init(SSLSocketImpl.java:604) >> >> >> >> at sun.security.ssl.SSLSocketImpl.<init>(http://SSLSocketImpl.java:572) >> >> >> >> at >> >> sun.security.ssl.SSLSocketFactoryImpl.createSocket(SSLSocketFactoryImpl.java:110) >> >> at >> >> org.apache.flink.fs.s3presto.shaded.org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:365) >> >> at >> >> org.apache.flink.fs.s3presto.shaded.org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:355) >> >> at >> >> org.apache.flink.fs.s3presto.shaded.com.amazonaws.http.conn.ssl.SdkTLSSocketFactory.connectSocket(SdkTLSSocketFactory.java:132) >> >> at >> >> org.apache.flink.fs.s3presto.shaded.org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) >> >> at >> >> org.apache.flink.fs.s3presto.shaded.org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:359) >> >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> >> at >> >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) >> >> at >> >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> >> at java.lang.reflect.Method.invoke(Method.java:498) >> >> at >> >> org.apache.flink.fs.s3presto.shaded.com.amazonaws.http.conn.ClientConnectionManagerFactory$Handler.invoke(ClientConnectionManagerFactory.java:76) >> >> at >> >> org.apache.flink.fs.s3presto.shaded.com.amazonaws.http.conn.$Proxy4.connect(Unknown >> >> Source) >> >> at >> >> org.apache.flink.fs.s3presto.shaded.org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381) >> >> at >> >> org.apache.flink.fs.s3presto.shaded.org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237) >> >> at >> >> org.apache.flink.fs.s3presto.shaded.org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) >> >> at >> >> org.apache.flink.fs.s3presto.shaded.org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) >> >> at >> >> org.apache.flink.fs.s3presto.shaded.org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) >> >> at >> >> org.apache.flink.fs.s3presto.shaded.org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56) >> >> at >> >> org.apache.flink.fs.s3presto.shaded.com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72) >> >> at >> >> org.apache.flink.fs.s3presto.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1189) >> >> at >> >> org.apache.flink.fs.s3presto.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1029) >> >> at >> >> org.apache.flink.fs.s3presto.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:741) >> >> at >> >> org.apache.flink.fs.s3presto.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:715) >> >> at >> >> org.apache.flink.fs.s3presto.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:697) >> >> at >> >> org.apache.flink.fs.s3presto.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:665) >> >> at >> >> org.apache.flink.fs.s3presto.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:647) >> >> at >> >> org.apache.flink.fs.s3presto.shaded.com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:511) >> >> at >> >> org.apache.flink.fs.s3presto.shaded.com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4227) >> >> at >> >> org.apache.flink.fs.s3presto.shaded.com.amazonaws.services.s3.AmazonS3Client.getBucketRegionViaHeadRequest(AmazonS3Client.java:4988) >> >> at >> >> org.apache.flink.fs.s3presto.shaded.com.amazonaws.services.s3.AmazonS3Client.fetchRegionFromCache(AmazonS3Client.java:4962) >> >> at >> >> org.apache.flink.fs.s3presto.shaded.com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4211) >> >> at >> >> org.apache.flink.fs.s3presto.shaded.com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4174) >> >> at >> >> org.apache.flink.fs.s3presto.shaded.com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:1253) >> >> at >> >> org.apache.flink.fs.s3presto.shaded.com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:1228) >> >> at >> >> org.apache.flink.fs.s3presto.shaded.com.facebook.presto.hive.PrestoS3FileSystem.lambda$getS3ObjectMetadata$2(PrestoS3FileSystem.java:559) >> >> at >> >> org.apache.flink.fs.s3presto.shaded.com.facebook.presto.hive.RetryDriver.run(RetryDriver.java:138) >> >> at >> >> org.apache.flink.fs.s3presto.shaded.com.facebook.presto.hive.PrestoS3FileSystem.getS3ObjectMetadata(PrestoS3FileSystem.java:556) >> >> at >> >> org.apache.flink.fs.s3presto.shaded.com.facebook.presto.hive.PrestoS3FileSystem.getFileStatus(PrestoS3FileSystem.java:307) >> >> at >> >> org.apache.flink.fs.s3presto.shaded.org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1426) >> >> at >> >> org.apache.flink.fs.s3presto.shaded.com.facebook.presto.hive.PrestoS3FileSystem.create(PrestoS3FileSystem.java:352) >> >> at >> >> org.apache.flink.fs.s3presto.shaded.org.apache.hadoop.fs.FileSystem.create(FileSystem.java:911) >> >> at >> >> org.apache.flink.fs.s3presto.shaded.org.apache.hadoop.fs.FileSystem.create(FileSystem.java:892) >> >> at >> >> org.apache.flink.fs.s3presto.shaded.org.apache.hadoop.fs.FileSystem.create(FileSystem.java:789) >> >> at >> >> org.apache.flink.fs.s3presto.shaded.org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.create(HadoopFileSystem.java:141) >> >> at >> >> org.apache.flink.fs.s3presto.shaded.org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.create(HadoopFileSystem.java:37) >> >> at >> >> org.apache.flink.core.fs.SafetyNetWrapperFileSystem.create(SafetyNetWrapperFileSystem.java:126) >> >> at >> >> org.apache.flink.core.fs.EntropyInjector.createEntropyAware(EntropyInjector.java:61) >> >> at >> >> org.apache.flink.runtime.state.filesystem.FsCheckpointStreamFactory$FsCheckpointStateOutputStream.createStream(FsCheckpointStreamFactory.java:350) >> >> at >> >> org.apache.flink.runtime.state.filesystem.FsCheckpointStreamFactory$FsCheckpointStateOutputStream.flush(FsCheckpointStreamFactory.java:228) >> >> at >> >> org.apache.flink.runtime.state.filesystem.FsCheckpointStreamFactory$FsCheckpointStateOutputStream.write(FsCheckpointStreamFactory.java:212) >> >> at java.io.DataOutputStream.write(DataOutputStream.java:107) >> >> at >> >> org.apache.flink.api.common.typeutils.TypeSerializerSerializationUtil.writeSerializersAndConfigsWithResilience(TypeSerializerSerializationUtil.java:161) >> >> at >> >> org.apache.flink.runtime.state.metainfo.StateMetaInfoSnapshotReadersWriters$CurrentWriterImpl.writeStateMetaInfoSnapshot(StateMetaInfoSnapshotReadersWriters.java:182) >> >> at >> >> org.apache.flink.runtime.state.OperatorBackendSerializationProxy.writeStateMetaInfoSnapshots(OperatorBackendSerializationProxy.java:85) >> >> at >> >> org.apache.flink.runtime.state.OperatorBackendSerializationProxy.write(OperatorBackendSerializationProxy.java:76) >> >> at >> >> org.apache.flink.runtime.state.DefaultOperatorStateBackend$1.performOperation(DefaultOperatorStateBackend.java:418) >> >> at >> >> org.apache.flink.runtime.state.DefaultOperatorStateBackend$1.performOperation(DefaultOperatorStateBackend.java:359) >> >> at >> >> org.apache.flink.runtime.io.async.AbstractAsyncCallableWithResources.call(AbstractAsyncCallableWithResources.java:75) >> >> at java.util.concurrent.FutureTask.run(FutureTask.java:266) >> >> at org.apache.flink.util.FutureUtil.runIfNotDoneAndGet(FutureUtil.java:50) >> >> ... 7 more >> >> 2018-11-02 23:29:32,983 INFO >> >> org.apache.flink.runtime.executiongraph.ExecutionGraph - Try to restart >> >> or fail the job ConnectedStreams maxwell.accounts >> >> (00000000000000000000000000000000) if no longer possible. >> >> >> >> ========================= >> >> >> >> On Thu, Nov 1, 2018 at 9:22 PM Hao Sun <ha...@zendesk.com> wrote: >> >>> >> >>> I am on Flink 1.6.2 (no Hadoop, in docker + K8S), using rocksdb and S3 >> >>> (presto) >> >>> I got this error when flink creating a checking point >> >>> >> >>> >> >>> =========== >> >>> 2018-11-02 04:00:55,011 INFO >> >>> org.apache.flink.runtime.executiongraph.ExecutionGraph - Job >> >>> ConnectedStreams maxwell.accounts (00000000000000000000000000000000) >> >>> switched from state RUNNING to FAILING. >> >>> AsynchronousException{java.lang.Exception: Could not materialize >> >>> checkpoint 130 for operator Source: KafkaSource(maxwell.accounts) -> >> >>> MaxwellFilter->Maxwell(maxwell.accounts) -> >> >>> FixedDelayWatermark(maxwell.accounts) -> >> >>> MaxwellFPSEvent->InfluxDBData(maxwell.accounts) -> Sink: >> >>> influxdbSink(maxwell.accounts) (1/1).} >> >>> at >> >>> org.apache.flink.streaming.runtime.tasks.StreamTask$AsyncCheckpointExceptionHandler.tryHandleCheckpointException(StreamTask.java:1153) >> >>> at >> >>> org.apache.flink.streaming.runtime.tasks.StreamTask$AsyncCheckpointRunnable.handleExecutionException(StreamTask.java:947) >> >>> at >> >>> org.apache.flink.streaming.runtime.tasks.StreamTask$AsyncCheckpointRunnable.run(StreamTask.java:884) >> >>> at >> >>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) >> >>> at java.util.concurrent.FutureTask.run(FutureTask.java:266) >> >>> at >> >>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) >> >>> at >> >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) >> >>> at java.lang.Thread.run(Thread.java:748) >> >>> Caused by: java.lang.Exception: Could not materialize checkpoint 130 for >> >>> operator Source: KafkaSource(maxwell.accounts) -> >> >>> MaxwellFilter->Maxwell(maxwell.accounts) -> >> >>> FixedDelayWatermark(maxwell.accounts) -> >> >>> MaxwellFPSEvent->InfluxDBData(maxwell.accounts) -> Sink: >> >>> influxdbSink(maxwell.accounts) (1/1). >> >>> at >> >>> org.apache.flink.streaming.runtime.tasks.StreamTask$AsyncCheckpointRunnable.handleExecutionException(StreamTask.java:942) >> >>> ... 6 more >> >>> Caused by: java.util.concurrent.ExecutionException: >> >>> java.lang.ExceptionInInitializerError >> >>> at java.util.concurrent.FutureTask.report(FutureTask.java:122) >> >>> at java.util.concurrent.FutureTask.get(FutureTask.java:192) >> >>> at >> >>> org.apache.flink.util.FutureUtil.runIfNotDoneAndGet(FutureUtil.java:53) >> >>> >> >>> at >> >>> org.apache.flink.streaming.api.operators.OperatorSnapshotFinalizer.<init>(http://OperatorSnapshotFinalizer.java:53) >> >>> >> >>> >> >>> at >> >>> org.apache.flink.streaming.runtime.tasks.StreamTask$AsyncCheckpointRunnable.run(StreamTask.java:853) >> >>> ... 5 more >> >>> Caused by: java.lang.ExceptionInInitializerError >> >>> >> >>> at sun.security.ssl.SSLSessionImpl.<init>(http://SSLSessionImpl.java:183) >> >>> at sun.security.ssl.SSLSessionImpl.<init>(http://SSLSessionImpl.java:148) >> >>> at >> >>> sun.security.ssl.SSLSessionImpl.<clinit>(http://SSLSessionImpl.java:79) >> >>> at sun.security.ssl.SSLSocketImpl.init(SSLSocketImpl.java:604) >> >>> at sun.security.ssl.SSLSocketImpl.<init>(http://SSLSocketImpl.java:572) >> >>> >> >>> >> >>> at >> >>> sun.security.ssl.SSLSocketFactoryImpl.createSocket(SSLSocketFactoryImpl.java:110) >> >>> at >> >>> org.apache.flink.fs.s3presto.shaded.org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:365) >> >>> at >> >>> org.apache.flink.fs.s3presto.shaded.org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:355) >> >>> at >> >>> org.apache.flink.fs.s3presto.shaded.com.amazonaws.http.conn.ssl.SdkTLSSocketFactory.connectSocket(SdkTLSSocketFactory.java:132) >> >>> at >> >>> org.apache.flink.fs.s3presto.shaded.org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) >> >>> at >> >>> org.apache.flink.fs.s3presto.shaded.org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:359) >> >>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> >>> at >> >>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) >> >>> at >> >>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> >>> at java.lang.reflect.Method.invoke(Method.java:498) >> >>> at >> >>> org.apache.flink.fs.s3presto.shaded.com.amazonaws.http.conn.ClientConnectionManagerFactory$Handler.invoke(ClientConnectionManagerFactory.java:76) >> >>> at >> >>> org.apache.flink.fs.s3presto.shaded.com.amazonaws.http.conn.$Proxy4.connect(Unknown >> >>> Source) >> >>> at >> >>> org.apache.flink.fs.s3presto.shaded.org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381) >> >>> at >> >>> org.apache.flink.fs.s3presto.shaded.org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237) >> >>> at >> >>> org.apache.flink.fs.s3presto.shaded.org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) >> >>> at >> >>> org.apache.flink.fs.s3presto.shaded.org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) >> >>> at >> >>> org.apache.flink.fs.s3presto.shaded.org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) >> >>> at >> >>> org.apache.flink.fs.s3presto.shaded.org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56) >> >>> at >> >>> org.apache.flink.fs.s3presto.shaded.com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72) >> >>> at >> >>> org.apache.flink.fs.s3presto.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1189) >> >>> at >> >>> org.apache.flink.fs.s3presto.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1029) >> >>> at >> >>> org.apache.flink.fs.s3presto.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:741) >> >>> at >> >>> org.apache.flink.fs.s3presto.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:715) >> >>> at >> >>> org.apache.flink.fs.s3presto.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:697) >> >>> at >> >>> org.apache.flink.fs.s3presto.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:665) >> >>> at >> >>> org.apache.flink.fs.s3presto.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:647) >> >>> at >> >>> org.apache.flink.fs.s3presto.shaded.com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:511) >> >>> at >> >>> org.apache.flink.fs.s3presto.shaded.com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4227) >> >>> at >> >>> org.apache.flink.fs.s3presto.shaded.com.amazonaws.services.s3.AmazonS3Client.getBucketRegionViaHeadRequest(AmazonS3Client.java:4988) >> >>> at >> >>> org.apache.flink.fs.s3presto.shaded.com.amazonaws.services.s3.AmazonS3Client.fetchRegionFromCache(AmazonS3Client.java:4962) >> >>> at >> >>> org.apache.flink.fs.s3presto.shaded.com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4211) >> >>> at >> >>> org.apache.flink.fs.s3presto.shaded.com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4174) >> >>> at >> >>> org.apache.flink.fs.s3presto.shaded.com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:1253) >> >>> at >> >>> org.apache.flink.fs.s3presto.shaded.com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:1228) >> >>> at >> >>> org.apache.flink.fs.s3presto.shaded.com.facebook.presto.hive.PrestoS3FileSystem.lambda$getS3ObjectMetadata$2(PrestoS3FileSystem.java:559) >> >>> at >> >>> org.apache.flink.fs.s3presto.shaded.com.facebook.presto.hive.RetryDriver.run(RetryDriver.java:138) >> >>> at >> >>> org.apache.flink.fs.s3presto.shaded.com.facebook.presto.hive.PrestoS3FileSystem.getS3ObjectMetadata(PrestoS3FileSystem.java:556) >> >>> at >> >>> org.apache.flink.fs.s3presto.shaded.com.facebook.presto.hive.PrestoS3FileSystem.getFileStatus(PrestoS3FileSystem.java:307) >> >>> at >> >>> org.apache.flink.fs.s3presto.shaded.org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1426) >> >>> at >> >>> org.apache.flink.fs.s3presto.shaded.com.facebook.presto.hive.PrestoS3FileSystem.create(PrestoS3FileSystem.java:352) >> >>> at >> >>> org.apache.flink.fs.s3presto.shaded.org.apache.hadoop.fs.FileSystem.create(FileSystem.java:911) >> >>> at >> >>> org.apache.flink.fs.s3presto.shaded.org.apache.hadoop.fs.FileSystem.create(FileSystem.java:892) >> >>> at >> >>> org.apache.flink.fs.s3presto.shaded.org.apache.hadoop.fs.FileSystem.create(FileSystem.java:789) >> >>> at >> >>> org.apache.flink.fs.s3presto.shaded.org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.create(HadoopFileSystem.java:141) >> >>> at >> >>> org.apache.flink.fs.s3presto.shaded.org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.create(HadoopFileSystem.java:37) >> >>> at >> >>> org.apache.flink.core.fs.SafetyNetWrapperFileSystem.create(SafetyNetWrapperFileSystem.java:126) >> >>> at >> >>> org.apache.flink.core.fs.EntropyInjector.createEntropyAware(EntropyInjector.java:61) >> >>> at >> >>> org.apache.flink.runtime.state.filesystem.FsCheckpointStreamFactory$FsCheckpointStateOutputStream.createStream(FsCheckpointStreamFactory.java:350) >> >>> at >> >>> org.apache.flink.runtime.state.filesystem.FsCheckpointStreamFactory$FsCheckpointStateOutputStream.flush(FsCheckpointStreamFactory.java:228) >> >>> at >> >>> org.apache.flink.runtime.state.filesystem.FsCheckpointStreamFactory$FsCheckpointStateOutputStream.write(FsCheckpointStreamFactory.java:212) >> >>> at java.io.DataOutputStream.write(DataOutputStream.java:107) >> >>> at >> >>> org.apache.flink.api.common.typeutils.TypeSerializerSerializationUtil.writeSerializersAndConfigsWithResilience(TypeSerializerSerializationUtil.java:161) >> >>> at >> >>> org.apache.flink.runtime.state.metainfo.StateMetaInfoSnapshotReadersWriters$CurrentWriterImpl.writeStateMetaInfoSnapshot(StateMetaInfoSnapshotReadersWriters.java:182) >> >>> at >> >>> org.apache.flink.runtime.state.OperatorBackendSerializationProxy.writeStateMetaInfoSnapshots(OperatorBackendSerializationProxy.java:85) >> >>> at >> >>> org.apache.flink.runtime.state.OperatorBackendSerializationProxy.write(OperatorBackendSerializationProxy.java:76) >> >>> at >> >>> org.apache.flink.runtime.state.DefaultOperatorStateBackend$1.performOperation(DefaultOperatorStateBackend.java:418) >> >>> at >> >>> org.apache.flink.runtime.state.DefaultOperatorStateBackend$1.performOperation(DefaultOperatorStateBackend.java:359) >> >>> at >> >>> org.apache.flink.runtime.io.async.AbstractAsyncCallableWithResources.call(AbstractAsyncCallableWithResources.java:75) >> >>> at java.util.concurrent.FutureTask.run(FutureTask.java:266) >> >>> at >> >>> org.apache.flink.util.FutureUtil.runIfNotDoneAndGet(FutureUtil.java:50) >> >>> ... 7 more >> >>> Caused by: java.security.ProviderException: Could not initialize NSS >> >>> >> >>> at sun.security.pkcs11.SunPKCS11.<init>(http://SunPKCS11.java:223) >> >>> at sun.security.pkcs11.SunPKCS11.<init>(http://SunPKCS11.java:103) >> >>> >> >>> >> >>> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) >> >>> at >> >>> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) >> >>> at >> >>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) >> >>> at java.lang.reflect.Constructor.newInstance(Constructor.java:423) >> >>> at sun.security.jca.ProviderConfig$2.run(ProviderConfig.java:224) >> >>> at sun.security.jca.ProviderConfig$2.run(ProviderConfig.java:206) >> >>> at java.security.AccessController.doPrivileged(Native Method) >> >>> at >> >>> sun.security.jca.ProviderConfig.doLoadProvider(ProviderConfig.java:206) >> >>> at sun.security.jca.ProviderConfig.getProvider(ProviderConfig.java:187) >> >>> at sun.security.jca.ProviderList.getProvider(ProviderList.java:233) >> >>> at sun.security.jca.ProviderList.getIndex(ProviderList.java:263) >> >>> at sun.security.jca.ProviderList.getProviderConfig(ProviderList.java:247) >> >>> at sun.security.jca.ProviderList.getProvider(ProviderList.java:253) >> >>> at java.security.Security.getProvider(Security.java:501) >> >>> >> >>> at >> >>> sun.security.ssl.SignatureAndHashAlgorithm.<clinit>(http://SignatureAndHashAlgorithm.java:415) >> >>> >> >>> >> >>> ... 72 more >> >>> Caused by: java.io.IOException: NSS is already initialized >> >>> at sun.security.pkcs11.Secmod.initialize(Secmod.java:173) >> >>> >> >>> at sun.security.pkcs11.SunPKCS11.<init>(http://SunPKCS11.java:218) >> >>> ... 88 more >> >>> >> >>> ===========
