Hi Ufuk, thanks for checking. I am using openJDK 1.8_171, I still have the same issue with presto.
- why checkpoint is not starting from 1? old chk stored in ZK caused it, I cleaned it up, but not very helpful - I switched to Flink + Hadoop28, and used hadoop s3, with no other changes, check pointing is working with the hadoop flavour. On Fri, Nov 9, 2018 at 2:02 PM Ufuk Celebi <u...@apache.org> wrote: > Hey Hao Sun, > > - Is this an intermittent failure or permanent? The logs indicate that > some checkpoints completed before the error occurs (e.g. checkpoint > numbers are greater than 1). > > - Which Java versions are you using? And which Java image? I've > Googled similar issues that seem to be related to the JVM, e.g. [1]. > > Best, > > Ufuk > > [1] > https://dev.lucee.org/t/could-not-initialize-class-sun-security-ssl-sslcontextimp/3972 > <https://dev.lucee.org/t/could-not-initialize-class-sun-security-ssl-sslcontextimp/3972> > > > On Thu, Nov 8, 2018 at 8:55 PM Hao Sun <ha...@zendesk.com> wrote: > > > > Thanks, any insight/help here is appreciated. > > > > On Thu, Nov 8, 2018 at 4:38 AM Dawid Wysakowicz <dwysakow...@apache.org> > wrote: > >> > >> Hi Hao, > >> > >> I am not sure, what might be wrong, but I've cc'ed Gary and Kostas who > were recently working with S3, maybe they will have some ideas. > >> > >> Best, > >> > >> Dawid > >> > >> On 03/11/2018 03:09, Hao Sun wrote: > >> > >> Same environment, new error. > >> > >> I can run the same docker image with my local Mac, but on K8S, this > gives me this error. > >> I can not think of any difference between local Docker and K8S Docker. > >> > >> Any hint will be helpful. Thanks > >> > >> ============ > >> > >> 2018-11-02 23:29:32,981 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Job > ConnectedStreams maxwell.accounts (00000000000000000000000000000000) > switched from state RUNNING to FAILING. > >> AsynchronousException{java.lang.Exception: Could not materialize > checkpoint 235 for operator Source: KafkaSource(maxwell.accounts) -> > MaxwellFilter->Maxwell(maxwell.accounts) -> > FixedDelayWatermark(maxwell.accounts) -> > MaxwellFPSEvent->InfluxDBData(maxwell.accounts) -> Sink: > influxdbSink(maxwell.accounts) (1/1).} > >> at > org.apache.flink.streaming.runtime.tasks.StreamTask$AsyncCheckpointExceptionHandler.tryHandleCheckpointException(StreamTask.java:1153) > >> at > org.apache.flink.streaming.runtime.tasks.StreamTask$AsyncCheckpointRunnable.handleExecutionException(StreamTask.java:947) > >> at > org.apache.flink.streaming.runtime.tasks.StreamTask$AsyncCheckpointRunnable.run(StreamTask.java:884) > >> at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > >> at java.util.concurrent.FutureTask.run(FutureTask.java:266) > >> at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > >> at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > >> at java.lang.Thread.run(Thread.java:748) > >> Caused by: java.lang.Exception: Could not materialize checkpoint 235 > for operator Source: KafkaSource(maxwell.accounts) -> > MaxwellFilter->Maxwell(maxwell.accounts) -> > FixedDelayWatermark(maxwell.accounts) -> > MaxwellFPSEvent->InfluxDBData(maxwell.accounts) -> Sink: > influxdbSink(maxwell.accounts) (1/1). > >> at > org.apache.flink.streaming.runtime.tasks.StreamTask$AsyncCheckpointRunnable.handleExecutionException(StreamTask.java:942) > >> ... 6 more > >> Caused by: java.util.concurrent.ExecutionException: > java.lang.NoClassDefFoundError: Could not initialize class > sun.security.ssl.SSLSessionImpl > >> at java.util.concurrent.FutureTask.report(FutureTask.java:122) > >> at java.util.concurrent.FutureTask.get(FutureTask.java:192) > >> at > org.apache.flink.util.FutureUtil.runIfNotDoneAndGet(FutureUtil.java:53) > >> > >> at > org.apache.flink.streaming.api.operators.OperatorSnapshotFinalizer.<init>( > http://OperatorSnapshotFinalizer.java:53 > <http://OperatorSnapshotFinalizer.java:53>) > >> > >> at > org.apache.flink.streaming.runtime.tasks.StreamTask$AsyncCheckpointRunnable.run(StreamTask.java:853) > >> ... 5 more > >> Caused by: java.lang.NoClassDefFoundError: Could not initialize class > sun.security.ssl.SSLSessionImpl > >> at sun.security.ssl.SSLSocketImpl.init(SSLSocketImpl.java:604) > >> > >> at sun.security.ssl.SSLSocketImpl.<init>(http://SSLSocketImpl.java:572 > <http://SSLSocketImpl.java:572>) > >> > >> at > sun.security.ssl.SSLSocketFactoryImpl.createSocket(SSLSocketFactoryImpl.java:110) > >> at > org.apache.flink.fs.s3presto.shaded.org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:365) > >> at > org.apache.flink.fs.s3presto.shaded.org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:355) > >> at > org.apache.flink.fs.s3presto.shaded.com.amazonaws.http.conn.ssl.SdkTLSSocketFactory.connectSocket(SdkTLSSocketFactory.java:132) > >> at > org.apache.flink.fs.s3presto.shaded.org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) > >> at > org.apache.flink.fs.s3presto.shaded.org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:359) > >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > >> at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > >> at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > >> at java.lang.reflect.Method.invoke(Method.java:498) > >> at > org.apache.flink.fs.s3presto.shaded.com.amazonaws.http.conn.ClientConnectionManagerFactory$Handler.invoke(ClientConnectionManagerFactory.java:76) > >> at > org.apache.flink.fs.s3presto.shaded.com.amazonaws.http.conn.$Proxy4.connect(Unknown > Source) > >> at > org.apache.flink.fs.s3presto.shaded.org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381) > >> at > org.apache.flink.fs.s3presto.shaded.org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237) > >> at > org.apache.flink.fs.s3presto.shaded.org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) > >> at > org.apache.flink.fs.s3presto.shaded.org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) > >> at > org.apache.flink.fs.s3presto.shaded.org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) > >> at > org.apache.flink.fs.s3presto.shaded.org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56) > >> at > org.apache.flink.fs.s3presto.shaded.com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72) > >> at > org.apache.flink.fs.s3presto.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1189) > >> at > org.apache.flink.fs.s3presto.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1029) > >> at > org.apache.flink.fs.s3presto.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:741) > >> at > org.apache.flink.fs.s3presto.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:715) > >> at > org.apache.flink.fs.s3presto.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:697) > >> at > org.apache.flink.fs.s3presto.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:665) > >> at > org.apache.flink.fs.s3presto.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:647) > >> at > org.apache.flink.fs.s3presto.shaded.com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:511) > >> at > org.apache.flink.fs.s3presto.shaded.com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4227) > >> at > org.apache.flink.fs.s3presto.shaded.com.amazonaws.services.s3.AmazonS3Client.getBucketRegionViaHeadRequest(AmazonS3Client.java:4988) > >> at > org.apache.flink.fs.s3presto.shaded.com.amazonaws.services.s3.AmazonS3Client.fetchRegionFromCache(AmazonS3Client.java:4962) > >> at > org.apache.flink.fs.s3presto.shaded.com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4211) > >> at > org.apache.flink.fs.s3presto.shaded.com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4174) > >> at > org.apache.flink.fs.s3presto.shaded.com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:1253) > >> at > org.apache.flink.fs.s3presto.shaded.com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:1228) > >> at > org.apache.flink.fs.s3presto.shaded.com.facebook.presto.hive.PrestoS3FileSystem.lambda$getS3ObjectMetadata$2(PrestoS3FileSystem.java:559) > >> at > org.apache.flink.fs.s3presto.shaded.com.facebook.presto.hive.RetryDriver.run(RetryDriver.java:138) > >> at > org.apache.flink.fs.s3presto.shaded.com.facebook.presto.hive.PrestoS3FileSystem.getS3ObjectMetadata(PrestoS3FileSystem.java:556) > >> at > org.apache.flink.fs.s3presto.shaded.com.facebook.presto.hive.PrestoS3FileSystem.getFileStatus(PrestoS3FileSystem.java:307) > >> at > org.apache.flink.fs.s3presto.shaded.org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1426) > >> at > org.apache.flink.fs.s3presto.shaded.com.facebook.presto.hive.PrestoS3FileSystem.create(PrestoS3FileSystem.java:352) > >> at > org.apache.flink.fs.s3presto.shaded.org.apache.hadoop.fs.FileSystem.create(FileSystem.java:911) > >> at > org.apache.flink.fs.s3presto.shaded.org.apache.hadoop.fs.FileSystem.create(FileSystem.java:892) > >> at > org.apache.flink.fs.s3presto.shaded.org.apache.hadoop.fs.FileSystem.create(FileSystem.java:789) > >> at > org.apache.flink.fs.s3presto.shaded.org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.create(HadoopFileSystem.java:141) > >> at > org.apache.flink.fs.s3presto.shaded.org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.create(HadoopFileSystem.java:37) > >> at > org.apache.flink.core.fs.SafetyNetWrapperFileSystem.create(SafetyNetWrapperFileSystem.java:126) > >> at > org.apache.flink.core.fs.EntropyInjector.createEntropyAware(EntropyInjector.java:61) > >> at > org.apache.flink.runtime.state.filesystem.FsCheckpointStreamFactory$FsCheckpointStateOutputStream.createStream(FsCheckpointStreamFactory.java:350) > >> at > org.apache.flink.runtime.state.filesystem.FsCheckpointStreamFactory$FsCheckpointStateOutputStream.flush(FsCheckpointStreamFactory.java:228) > >> at > org.apache.flink.runtime.state.filesystem.FsCheckpointStreamFactory$FsCheckpointStateOutputStream.write(FsCheckpointStreamFactory.java:212) > >> at java.io.DataOutputStream.write(DataOutputStream.java:107) > >> at > org.apache.flink.api.common.typeutils.TypeSerializerSerializationUtil.writeSerializersAndConfigsWithResilience(TypeSerializerSerializationUtil.java:161) > >> at > org.apache.flink.runtime.state.metainfo.StateMetaInfoSnapshotReadersWriters$CurrentWriterImpl.writeStateMetaInfoSnapshot(StateMetaInfoSnapshotReadersWriters.java:182) > >> at > org.apache.flink.runtime.state.OperatorBackendSerializationProxy.writeStateMetaInfoSnapshots(OperatorBackendSerializationProxy.java:85) > >> at > org.apache.flink.runtime.state.OperatorBackendSerializationProxy.write(OperatorBackendSerializationProxy.java:76) > >> at > org.apache.flink.runtime.state.DefaultOperatorStateBackend$1.performOperation(DefaultOperatorStateBackend.java:418) > >> at > org.apache.flink.runtime.state.DefaultOperatorStateBackend$1.performOperation(DefaultOperatorStateBackend.java:359) > >> at > org.apache.flink.runtime.io.async.AbstractAsyncCallableWithResources.call(AbstractAsyncCallableWithResources.java:75) > >> at java.util.concurrent.FutureTask.run(FutureTask.java:266) > >> at > org.apache.flink.util.FutureUtil.runIfNotDoneAndGet(FutureUtil.java:50) > >> ... 7 more > >> 2018-11-02 23:29:32,983 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Try to restart or > fail the job ConnectedStreams maxwell.accounts > (00000000000000000000000000000000) if no longer possible. > >> > >> ========================= > >> > >> On Thu, Nov 1, 2018 at 9:22 PM Hao Sun <ha...@zendesk.com> wrote: > >>> > >>> I am on Flink 1.6.2 (no Hadoop, in docker + K8S), using rocksdb and S3 > (presto) > >>> I got this error when flink creating a checking point > >>> > >>> > >>> =========== > >>> 2018-11-02 04:00:55,011 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Job > ConnectedStreams maxwell.accounts (00000000000000000000000000000000) > switched from state RUNNING to FAILING. > >>> AsynchronousException{java.lang.Exception: Could not materialize > checkpoint 130 for operator Source: KafkaSource(maxwell.accounts) -> > MaxwellFilter->Maxwell(maxwell.accounts) -> > FixedDelayWatermark(maxwell.accounts) -> > MaxwellFPSEvent->InfluxDBData(maxwell.accounts) -> Sink: > influxdbSink(maxwell.accounts) (1/1).} > >>> at > org.apache.flink.streaming.runtime.tasks.StreamTask$AsyncCheckpointExceptionHandler.tryHandleCheckpointException(StreamTask.java:1153) > >>> at > org.apache.flink.streaming.runtime.tasks.StreamTask$AsyncCheckpointRunnable.handleExecutionException(StreamTask.java:947) > >>> at > org.apache.flink.streaming.runtime.tasks.StreamTask$AsyncCheckpointRunnable.run(StreamTask.java:884) > >>> at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > >>> at java.util.concurrent.FutureTask.run(FutureTask.java:266) > >>> at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > >>> at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > >>> at java.lang.Thread.run(Thread.java:748) > >>> Caused by: java.lang.Exception: Could not materialize checkpoint 130 > for operator Source: KafkaSource(maxwell.accounts) -> > MaxwellFilter->Maxwell(maxwell.accounts) -> > FixedDelayWatermark(maxwell.accounts) -> > MaxwellFPSEvent->InfluxDBData(maxwell.accounts) -> Sink: > influxdbSink(maxwell.accounts) (1/1). > >>> at > org.apache.flink.streaming.runtime.tasks.StreamTask$AsyncCheckpointRunnable.handleExecutionException(StreamTask.java:942) > >>> ... 6 more > >>> Caused by: java.util.concurrent.ExecutionException: > java.lang.ExceptionInInitializerError > >>> at java.util.concurrent.FutureTask.report(FutureTask.java:122) > >>> at java.util.concurrent.FutureTask.get(FutureTask.java:192) > >>> at > org.apache.flink.util.FutureUtil.runIfNotDoneAndGet(FutureUtil.java:53) > >>> > >>> at > org.apache.flink.streaming.api.operators.OperatorSnapshotFinalizer.<init>( > http://OperatorSnapshotFinalizer.java:53 > <http://OperatorSnapshotFinalizer.java:53>) > >>> > >>> > >>> at > org.apache.flink.streaming.runtime.tasks.StreamTask$AsyncCheckpointRunnable.run(StreamTask.java:853) > >>> ... 5 more > >>> Caused by: java.lang.ExceptionInInitializerError > >>> > >>> at sun.security.ssl.SSLSessionImpl.<init>( > http://SSLSessionImpl.java:183 > <http://SSLSessionImpl.java:183>) > >>> at sun.security.ssl.SSLSessionImpl.<init>( > http://SSLSessionImpl.java:148 > <http://SSLSessionImpl.java:148>) > >>> at sun.security.ssl.SSLSessionImpl.<clinit>( > http://SSLSessionImpl.java:79 > <http://SSLSessionImpl.java:79>) > >>> at sun.security.ssl.SSLSocketImpl.init(SSLSocketImpl.java:604) > >>> at sun.security.ssl.SSLSocketImpl.<init>(http://SSLSocketImpl.java:572 > <http://SSLSocketImpl.java:572>) > >>> > >>> > >>> at > sun.security.ssl.SSLSocketFactoryImpl.createSocket(SSLSocketFactoryImpl.java:110) > >>> at > org.apache.flink.fs.s3presto.shaded.org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:365) > >>> at > org.apache.flink.fs.s3presto.shaded.org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:355) > >>> at > org.apache.flink.fs.s3presto.shaded.com.amazonaws.http.conn.ssl.SdkTLSSocketFactory.connectSocket(SdkTLSSocketFactory.java:132) > >>> at > org.apache.flink.fs.s3presto.shaded.org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) > >>> at > org.apache.flink.fs.s3presto.shaded.org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:359) > >>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > >>> at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > >>> at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > >>> at java.lang.reflect.Method.invoke(Method.java:498) > >>> at > org.apache.flink.fs.s3presto.shaded.com.amazonaws.http.conn.ClientConnectionManagerFactory$Handler.invoke(ClientConnectionManagerFactory.java:76) > >>> at > org.apache.flink.fs.s3presto.shaded.com.amazonaws.http.conn.$Proxy4.connect(Unknown > Source) > >>> at > org.apache.flink.fs.s3presto.shaded.org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381) > >>> at > org.apache.flink.fs.s3presto.shaded.org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237) > >>> at > org.apache.flink.fs.s3presto.shaded.org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) > >>> at > org.apache.flink.fs.s3presto.shaded.org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) > >>> at > org.apache.flink.fs.s3presto.shaded.org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) > >>> at > org.apache.flink.fs.s3presto.shaded.org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56) > >>> at > org.apache.flink.fs.s3presto.shaded.com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72) > >>> at > org.apache.flink.fs.s3presto.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1189) > >>> at > org.apache.flink.fs.s3presto.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1029) > >>> at > org.apache.flink.fs.s3presto.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:741) > >>> at > org.apache.flink.fs.s3presto.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:715) > >>> at > org.apache.flink.fs.s3presto.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:697) > >>> at > org.apache.flink.fs.s3presto.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:665) > >>> at > org.apache.flink.fs.s3presto.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:647) > >>> at > org.apache.flink.fs.s3presto.shaded.com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:511) > >>> at > org.apache.flink.fs.s3presto.shaded.com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4227) > >>> at > org.apache.flink.fs.s3presto.shaded.com.amazonaws.services.s3.AmazonS3Client.getBucketRegionViaHeadRequest(AmazonS3Client.java:4988) > >>> at > org.apache.flink.fs.s3presto.shaded.com.amazonaws.services.s3.AmazonS3Client.fetchRegionFromCache(AmazonS3Client.java:4962) > >>> at > org.apache.flink.fs.s3presto.shaded.com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4211) > >>> at > org.apache.flink.fs.s3presto.shaded.com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4174) > >>> at > org.apache.flink.fs.s3presto.shaded.com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:1253) > >>> at > org.apache.flink.fs.s3presto.shaded.com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:1228) > >>> at > org.apache.flink.fs.s3presto.shaded.com.facebook.presto.hive.PrestoS3FileSystem.lambda$getS3ObjectMetadata$2(PrestoS3FileSystem.java:559) > >>> at > org.apache.flink.fs.s3presto.shaded.com.facebook.presto.hive.RetryDriver.run(RetryDriver.java:138) > >>> at > org.apache.flink.fs.s3presto.shaded.com.facebook.presto.hive.PrestoS3FileSystem.getS3ObjectMetadata(PrestoS3FileSystem.java:556) > >>> at > org.apache.flink.fs.s3presto.shaded.com.facebook.presto.hive.PrestoS3FileSystem.getFileStatus(PrestoS3FileSystem.java:307) > >>> at > org.apache.flink.fs.s3presto.shaded.org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1426) > >>> at > org.apache.flink.fs.s3presto.shaded.com.facebook.presto.hive.PrestoS3FileSystem.create(PrestoS3FileSystem.java:352) > >>> at > org.apache.flink.fs.s3presto.shaded.org.apache.hadoop.fs.FileSystem.create(FileSystem.java:911) > >>> at > org.apache.flink.fs.s3presto.shaded.org.apache.hadoop.fs.FileSystem.create(FileSystem.java:892) > >>> at > org.apache.flink.fs.s3presto.shaded.org.apache.hadoop.fs.FileSystem.create(FileSystem.java:789) > >>> at > org.apache.flink.fs.s3presto.shaded.org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.create(HadoopFileSystem.java:141) > >>> at > org.apache.flink.fs.s3presto.shaded.org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.create(HadoopFileSystem.java:37) > >>> at > org.apache.flink.core.fs.SafetyNetWrapperFileSystem.create(SafetyNetWrapperFileSystem.java:126) > >>> at > org.apache.flink.core.fs.EntropyInjector.createEntropyAware(EntropyInjector.java:61) > >>> at > org.apache.flink.runtime.state.filesystem.FsCheckpointStreamFactory$FsCheckpointStateOutputStream.createStream(FsCheckpointStreamFactory.java:350) > >>> at > org.apache.flink.runtime.state.filesystem.FsCheckpointStreamFactory$FsCheckpointStateOutputStream.flush(FsCheckpointStreamFactory.java:228) > >>> at > org.apache.flink.runtime.state.filesystem.FsCheckpointStreamFactory$FsCheckpointStateOutputStream.write(FsCheckpointStreamFactory.java:212) > >>> at java.io.DataOutputStream.write(DataOutputStream.java:107) > >>> at > org.apache.flink.api.common.typeutils.TypeSerializerSerializationUtil.writeSerializersAndConfigsWithResilience(TypeSerializerSerializationUtil.java:161) > >>> at > org.apache.flink.runtime.state.metainfo.StateMetaInfoSnapshotReadersWriters$CurrentWriterImpl.writeStateMetaInfoSnapshot(StateMetaInfoSnapshotReadersWriters.java:182) > >>> at > org.apache.flink.runtime.state.OperatorBackendSerializationProxy.writeStateMetaInfoSnapshots(OperatorBackendSerializationProxy.java:85) > >>> at > org.apache.flink.runtime.state.OperatorBackendSerializationProxy.write(OperatorBackendSerializationProxy.java:76) > >>> at > org.apache.flink.runtime.state.DefaultOperatorStateBackend$1.performOperation(DefaultOperatorStateBackend.java:418) > >>> at > org.apache.flink.runtime.state.DefaultOperatorStateBackend$1.performOperation(DefaultOperatorStateBackend.java:359) > >>> at > org.apache.flink.runtime.io.async.AbstractAsyncCallableWithResources.call(AbstractAsyncCallableWithResources.java:75) > >>> at java.util.concurrent.FutureTask.run(FutureTask.java:266) > >>> at > org.apache.flink.util.FutureUtil.runIfNotDoneAndGet(FutureUtil.java:50) > >>> ... 7 more > >>> Caused by: java.security.ProviderException: Could not initialize NSS > >>> > >>> at sun.security.pkcs11.SunPKCS11.<init>(http://SunPKCS11.java:223 > <http://SunPKCS11.java:223>) > >>> at sun.security.pkcs11.SunPKCS11.<init>(http://SunPKCS11.java:103 > <http://SunPKCS11.java:103>) > >>> > >>> > >>> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native > Method) > >>> at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > >>> at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > >>> at java.lang.reflect.Constructor.newInstance(Constructor.java:423) > >>> at sun.security.jca.ProviderConfig$2.run(ProviderConfig.java:224) > >>> at sun.security.jca.ProviderConfig$2.run(ProviderConfig.java:206) > >>> at java.security.AccessController.doPrivileged(Native Method) > >>> at > sun.security.jca.ProviderConfig.doLoadProvider(ProviderConfig.java:206) > >>> at sun.security.jca.ProviderConfig.getProvider(ProviderConfig.java:187) > >>> at sun.security.jca.ProviderList.getProvider(ProviderList.java:233) > >>> at sun.security.jca.ProviderList.getIndex(ProviderList.java:263) > >>> at > sun.security.jca.ProviderList.getProviderConfig(ProviderList.java:247) > >>> at sun.security.jca.ProviderList.getProvider(ProviderList.java:253) > >>> at java.security.Security.getProvider(Security.java:501) > >>> > >>> at sun.security.ssl.SignatureAndHashAlgorithm.<clinit>( > http://SignatureAndHashAlgorithm.java:415 > <http://SignatureAndHashAlgorithm.java:415>) > >>> > >>> > >>> ... 72 more > >>> Caused by: java.io.IOException: NSS is already initialized > >>> at sun.security.pkcs11.Secmod.initialize(Secmod.java:173) > >>> > >>> at sun.security.pkcs11.SunPKCS11.<init>(http://SunPKCS11.java:218 > <http://SunPKCS11.java:218>) > >>> ... 88 more > >>> > >>> =========== >
