Hi Vishwas, Sorry for the late response. Are you still facing the issue?
I have no experience with EMC ECS, but the exception suggests an issue with the host name: 1378 Caused by: java.net.UnknownHostException: aip-featuretoolkit.SU73ECSG1P1d.***.COM 1379 at java.net.InetAddress.getAllByName0(InetAddress.java:1280) 1380 at java.net.InetAddress.getAllByName(InetAddress.java:1192) 1381 at java.net.InetAddress.getAllByName(InetAddress.java:1126) 1382 at org.apache.flink.fs.s3base.shaded.com.amazonaws.SystemDefaultDnsResolver.resolve(SystemDefaultDnsResolver.java:27) Best, Fabian Am Di., 25. Juni 2019 um 03:26 Uhr schrieb Vishwas Siravara < vsirav...@gmail.com>: > Hi, > I am using flink version 1.7.2 , I am trying to use S3 like object > storage EMC ECS( > https://www.emc.com/techpubs/ecs/ecs_s3_supported_features-1.htm). Not > all S3 apis are supported by EMC ESC according to this document. Here > is my config > > s3.endpoint: SU73ECSG1P1d.***.COM > s3.access-key: vdna_np_user > security.ssl.rest.enabled: false > web.timeout: 10000 > s3.secret-key: J*** > > I can access this bucket from s3cmd client. > > I set the state backend from my scala application > env.setStateBackend(new > FsStateBackend("s3://aip-featuretoolkit/checkpoints/")) > > However when I run my application I get this exception : > > ClientException: Unable to execute HTTP request: > aip-featuretoolkit.SU73ECSG1P1d.VISA.COM: Unable to execute HTTP > request: aip-featuretoolkit.SU73ECSG1P1d.VISA.COM > 1336 at > > org.apache.flink.runtime.checkpoint.CheckpointCoordinator.<init>(CheckpointCoordinator.java:255) > 1337 at > > org.apache.flink.runtime.executiongraph.ExecutionGraph.enableCheckpointing(ExecutionGraph.java:498) > 1338 at > > org.apache.flink.runtime.executiongraph.ExecutionGraphBuilder.buildGraph(ExecutionGraphBuilder.java:345) > 1339 at > > org.apache.flink.runtime.executiongraph.ExecutionGraphBuilder.buildGraph(ExecutionGraphBuilder.java:100) > 1340 at > > org.apache.flink.runtime.jobmaster.JobMaster.createExecutionGraph(JobMaster.java:1173) > 1341 at > > org.apache.flink.runtime.jobmaster.JobMaster.createAndRestoreExecutionGraph(JobMaster.java:1153) > 1342 at > org.apache.flink.runtime.jobmaster.JobMaster.<init>(JobMaster.java:296) > 1343 at > > org.apache.flink.runtime.jobmaster.JobManagerRunner.<init>(JobManagerRunner.java:157) > 1344 ... 10 more > 1345 Caused by: > > org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.s3a.AWSClientIOException: > doesBucketExist on aip-featuretoolkit: > org.apache.flink.fs.s3base.shaded.co > m.amazonaws.SdkClientException: Unable to execute HTTP request: > aip-featuretoolkit.SU73ECSG1P1d.VISA.COM: Unable to execute HTTP > request: aip-featuretoolkit.SU73ECSG1P1 d.VISA.COM > 1346 at > > org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:177) > 1347 at > > org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:111) > 1348 at > > org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.s3a.Invoker.lambda$retry$3(Invoker.java:260) > 1349 at > > org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:317) > 1350 at > > org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.s3a.Invoker.retry(Invoker.java:256) > 1351 at > > org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.s3a.Invoker.retry(Invoker.java:231) > 1352 at > > org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.s3a.S3AFileSystem.verifyBucketExists(S3AFileSystem.java:372) > 1353 at > > org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:308) > 1354 at > > org.apache.flink.fs.s3.common.AbstractS3FileSystemFactory.create(AbstractS3FileSystemFactory.java:125) > 1355 at > > org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:395) > 1356 at > org.apache.flink.core.fs.FileSystem.get(FileSystem.java:318) > 1357 at > org.apache.flink.core.fs.Path.getFileSystem(Path.java:298) > 1358 at > > org.apache.flink.runtime.state.filesystem.FsCheckpointStorage.<init>(FsCheckpointStorage.java:58) > 1359 at > > org.apache.flink.runtime.state.filesystem.FsStateBackend.createCheckpointStorage(FsStateBackend.java:444) > 1360 at > > org.apache.flink.runtime.checkpoint.CheckpointCoordinator.<init>(CheckpointCoordinator.java:249) > 1361 ... 17 more > 1362 Caused by: > org.apache.flink.fs.s3base.shaded.com.amazonaws.SdkClientException: > Unable to execute HTTP request: > aip-featuretoolkit.SU73ECSG1P1d.VISA.COM > 1363 at > > org.apache.flink.fs.s3base.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleRetryableException(AmazonHttpClient.java:1114) > 1364 at > > org.apache.flink.fs.s3base.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1064) > 1365 at > > org.apache.flink.fs.s3base.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:743) > 1366 at > > org.apache.flink.fs.s3base.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:717) > 1367 at > > org.apache.flink.fs.s3base.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699) > 1368 at > > org.apache.flink.fs.s3base.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667) > 1369 at > > org.apache.flink.fs.s3base.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649) > 1370 at > > org.apache.flink.fs.s3base.shaded.com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513) > 1371 at > > org.apache.flink.fs.s3base.shaded.com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4325) > 1372 at > > org.apache.flink.fs.s3base.shaded.com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4272) > 1373 at > > org.apache.flink.fs.s3base.shaded.com.amazonaws.services.s3.AmazonS3Client.headBucket(AmazonS3Client.java:1337) > 1374 at > > org.apache.flink.fs.s3base.shaded.com.amazonaws.services.s3.AmazonS3Client.doesBucketExist(AmazonS3Client.java:1277) > 1375 at > > org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$verifyBucketExists$1(S3AFileSystem.java:373) > 1376 at > > org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:109) > 1377 ... 30 more > 1378 Caused by: java.net.UnknownHostException: > aip-featuretoolkit.SU73ECSG1P1d.VISA.COM > 1379 at > java.net.InetAddress.getAllByName0(InetAddress.java:1280) > 1380 at java.net.InetAddress.getAllByName(InetAddress.java:1192) > 1381 at java.net.InetAddress.getAllByName(InetAddress.java:1126) > 1382 at > > org.apache.flink.fs.s3base.shaded.com.amazonaws.SystemDefaultDnsResolver.resolve(SystemDefaultDnsResolver.java:27) > 1383 at > > org.apache.flink.fs.s3base.shaded.com.amazonaws.http.DelegatingDnsResolver.resolve(DelegatingDnsResolver.java:38) > 1384 at > > org.apache.flink.fs.s3base.shaded.org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:112) > 1385 at > > org.apache.flink.fs.s3base.shaded.org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:359) > 1386 at sun.reflect.GeneratedMethodAccessor19.invoke(Unknown > Source) > 1387 at > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > 1388 at java.lang.reflect.Method.invoke(Method.java:498) > 1389 at > > org.apache.flink.fs.s3base.shaded.com.amazonaws.http.conn.ClientConnectionManagerFactory$Handler.invoke(ClientConnectionManagerFactory.java:76) > 1390 at > > org.apache.flink.fs.s3base.shaded.com.amazonaws.http.conn.$Proxy45.connect(Unknown > Source) > 1391 at > > org.apache.flink.fs.s3base.shaded.org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381) > 1392 at > > org.apache.flink.fs.s3base.shaded.org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237) > 1393 at > > org.apache.flink.fs.s3base.shaded.org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) > 1394 at > > org.apache.flink.fs.s3base.shaded.org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) > 1395 at > > org.apache.flink.fs.s3base.shaded.org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) > 1396 at > > org.apache.flink.fs.s3base.shaded.org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56) > 1397 at > > org.apache.flink.fs.s3base.shaded.com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72) > 1398 at > > org.apache.flink.fs.s3base.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1236) > 1399 at > > org.apache.flink.fs.s3base.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1056) > > Is this backend compatible with flink ? > > Thanks, > Vishwas >
