Hello, I am trying to upgrade our job from flink 1.4.2 to 1.7.1 but I keep running into timeouts after submitting the job.
The flink job runs on our hadoop cluster and starts using Yarn. Relevant config options seem to be: jobmanager.rpc.port: 55501 recovery.jobmanager.port: 55502 yarn.application-master.port: 55503 blob.server.port: 55504 I've seen the following behavior: - Using the same flink-conf.yaml as we used in 1.4.2: 1.5.6 / 1.6.3 / 1.7.1 all versions timeout while 1.4.2 works. - Using 1.5.6 with "mode: legacy" (to switch off flip-6) works - Using 1.7.1 with "mode: legacy" gives timeout (I assume this option was removed but the documentation is outdated? https://ci.apache.org/projects/flink/flink-docs-stable/ops/config.html#legacy ) When the timeout happens I get the following stacktrace: INFO class java.time.Instant does not contain a getter for field seconds 2019-02-18T10:16:56.815+01:00 INFO class com.bol.fin_hdp.cm1.domain.Cm1Transportable does not contain a getter for field globalId 2019-02-18T10:16:56.815+01:00 INFO Submitting job 5af931bcef395a78b5af2b97e92dcffe (detached: false). 2019-02-18T10:16:57.182+01:00 INFO ------------------------------------------------------------ 2019-02-18T10:29:27.527+01:00 INFO The program finished with the following exception: 2019-02-18T10:29:27.564+01:00 INFO org.apache.flink.client.program.ProgramInvocationException: The main method caused an error. 2019-02-18T10:29:27.601+01:00 INFO at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:545) 2019-02-18T10:29:27.638+01:00 INFO at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:420) 2019-02-18T10:29:27.675+01:00 INFO at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:404) 2019-02-18T10:29:27.711+01:00 INFO at org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:798) 2019-02-18T10:29:27.747+01:00 INFO at org.apache.flink.client.cli.CliFrontend.runProgram(CliFrontend.java:289) 2019-02-18T10:29:27.784+01:00 INFO at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:215) 2019-02-18T10:29:27.820+01:00 INFO at org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1035) 2019-02-18T10:29:27.857+01:00 INFO at org.apache.flink.client.cli.CliFrontend.lambda$main$9(CliFrontend.java:1111) 2019-02-18T10:29:27.893+01:00 INFO at java.security.AccessController.doPrivileged(Native Method) 2019-02-18T10:29:27.929+01:00 INFO at javax.security.auth.Subject.doAs(Subject.java:422) 2019-02-18T10:29:27.968+01:00 INFO at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1754) 2019-02-18T10:29:28.004+01:00 INFO at org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41) 2019-02-18T10:29:28.040+01:00 INFO at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1111) 2019-02-18T10:29:28.075+01:00 INFO Caused by: java.lang.RuntimeException: org.apache.flink.client.program.ProgramInvocationException: Could not retrieve the execution result. 2019-02-18T10:29:28.110+01:00 INFO at com.bol.fin_hdp.job.starter.IntervalJobStarter.startJob(IntervalJobStarter.java:43) 2019-02-18T10:29:28.146+01:00 INFO at com.bol.fin_hdp.job.starter.IntervalJobStarter.startJobWithConfig(IntervalJobStarter.java:32) 2019-02-18T10:29:28.182+01:00 INFO at com.bol.fin_hdp.Main.main(Main.java:8) 2019-02-18T10:29:28.217+01:00 INFO at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 2019-02-18T10:29:28.253+01:00 INFO at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 2019-02-18T10:29:28.289+01:00 INFO at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 2019-02-18T10:29:28.325+01:00 INFO at java.lang.reflect.Method.invoke(Method.java:498) 2019-02-18T10:29:28.363+01:00 INFO at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:528) 2019-02-18T10:29:28.400+01:00 INFO ... 12 more 2019-02-18T10:29:28.436+01:00 INFO Caused by: org.apache.flink.client.program.ProgramInvocationException: Could not retrieve the execution result. 2019-02-18T10:29:28.473+01:00 INFO at org.apache.flink.client.program.rest.RestClusterClient.submitJob(RestClusterClient.java:258) 2019-02-18T10:29:28.509+01:00 INFO at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:464) 2019-02-18T10:29:28.544+01:00 INFO at org.apache.flink.streaming.api.environment.StreamContextEnvironment.execute(StreamContextEnvironment.java:66) 2019-02-18T10:29:28.581+01:00 INFO at com.bol.fin_hdp.cm1.job.Job.execute(Job.java:54) 2019-02-18T10:29:28.617+01:00 INFO at com.bol.fin_hdp.job.starter.IntervalJobStarter.startJob(IntervalJobStarter.java:41) 2019-02-18T10:29:28.654+01:00 INFO ... 19 more 2019-02-18T10:29:28.693+01:00 INFO Caused by: org.apache.flink.runtime.client.JobSubmissionException: Failed to submit JobGraph. 2019-02-18T10:29:28.730+01:00 INFO at org.apache.flink.client.program.rest.RestClusterClient.lambda$submitJob$8(RestClusterClient.java:371) 2019-02-18T10:29:28.766+01:00 INFO at java.util.concurrent.CompletableFuture.uniExceptionally(CompletableFuture.java:870) 2019-02-18T10:29:28.803+01:00 INFO at java.util.concurrent.CompletableFuture$UniExceptionally.tryFire(CompletableFuture.java:852) 2019-02-18T10:29:28.839+01:00 INFO at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) 2019-02-18T10:29:28.876+01:00 INFO at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977) 2019-02-18T10:29:28.912+01:00 INFO at org.apache.flink.runtime.concurrent.FutureUtils.lambda$retryOperationWithDelay$5(FutureUtils.java:216) 2019-02-18T10:29:28.948+01:00 INFO at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) 2019-02-18T10:29:28.986+01:00 INFO at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) 2019-02-18T10:29:29.023+01:00 INFO at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) 2019-02-18T10:29:29.060+01:00 INFO at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977) 2019-02-18T10:29:29.096+01:00 INFO at org.apache.flink.runtime.rest.RestClient.lambda$submitRequest$1(RestClient.java:301) 2019-02-18T10:29:29.133+01:00 INFO at org.apache.flink.shaded.netty4.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:680) 2019-02-18T10:29:29.169+01:00 INFO at org.apache.flink.shaded.netty4.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:603) 2019-02-18T10:29:29.206+01:00 INFO at org.apache.flink.shaded.netty4.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:563) 2019-02-18T10:29:29.242+01:00 INFO at org.apache.flink.shaded.netty4.io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:424) 2019-02-18T10:29:29.278+01:00 INFO at org.apache.flink.shaded.netty4.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe$1.run(AbstractNioChannel.java:214) 2019-02-18T10:29:29.315+01:00 INFO at org.apache.flink.shaded.netty4.io.netty.util.concurrent.PromiseTask$RunnableAdapter.call(PromiseTask.java:38) 2019-02-18T10:29:29.352+01:00 INFO at org.apache.flink.shaded.netty4.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:120) 2019-02-18T10:29:29.388+01:00 INFO at org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:357) 2019-02-18T10:29:29.424+01:00 INFO at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:357) 2019-02-18T10:29:29.460+01:00 INFO at org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111) 2019-02-18T10:29:29.496+01:00 INFO at org.apache.flink.shaded.netty4.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137) 2019-02-18T10:29:29.532+01:00 INFO at java.lang.Thread.run(Thread.java:748) 2019-02-18T10:29:29.569+01:00 INFO Caused by: org.apache.flink.runtime.concurrent.FutureUtils$RetryException: Could not complete the operation. Number of retries has been exhausted. 2019-02-18T10:29:29.606+01:00 INFO at org.apache.flink.runtime.concurrent.FutureUtils.lambda$retryOperationWithDelay$5(FutureUtils.java:213) 2019-02-18T10:29:29.643+01:00 INFO ... 17 more 2019-02-18T10:29:29.680+01:00 INFO Caused by: java.util.concurrent.CompletionException: org.apache.flink.shaded.netty4.io.netty.channel.ConnectTimeoutException: connection timed out: shd-hdp-b-slave-01... 2019-02-18T10:29:29.717+01:00 INFO at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292) 2019-02-18T10:29:29.753+01:00 INFO at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308) 2019-02-18T10:29:29.789+01:00 INFO at java.util.concurrent.CompletableFuture.uniCompose(CompletableFuture.java:943) 2019-02-18T10:29:29.826+01:00 INFO at java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:926) 2019-02-18T10:29:29.862+01:00 INFO ... 15 more 2019-02-18T10:29:29.898+01:00 INFO Caused by: org.apache.flink.shaded.netty4.io.netty.channel.ConnectTimeoutException: connection timed out: shd-hdp-b-slave-017.example.com/some.ip.address:46500 2019-02-18T10:29:29.934+01:00 INFO at org.apache.flink.shaded.netty4.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe$1.run(AbstractNioChannel.java:212) 2019-02-18T10:29:29.970+01:00 INFO ... 7 more Does anyone have tips how to debug this or what configuration changes I need to make?
