[
https://issues.apache.org/jira/browse/FLINK-17750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17115877#comment-17115877
]
Yang Wang commented on FLINK-17750:
-----------------------------------
The test failed with timed out in {{DispatcherGateway.requestJob}}. If we
increase the timeout, even there is something wrong with the zookeeper and the
job restarts, it should finally turn into running. And then the Flink client
could get the job result correctly.
You could find the job has been restarted and finally turning into running in
the log[1].
I admin that increasing the timeout of zookeeper should be done first and i
also suggest to increase the akka ask timeout. It could help to reduce the
failure further.
[1]. [^jobmanager.log]
> YARNHighAvailabilityITCase.testKillYarnSessionClusterEntrypoint failed on
> azure
> -------------------------------------------------------------------------------
>
> Key: FLINK-17750
> URL: https://issues.apache.org/jira/browse/FLINK-17750
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
> Affects Versions: 1.11.0
> Reporter: Roman Khachatryan
> Assignee: Till Rohrmann
> Priority: Critical
> Labels: test-stability
> Fix For: 1.11.0
>
>
> [https://dev.azure.com/khachatryanroman/810e80cc-0656-4d3c-9d8c-186764456a01/_apis/build/builds/6/logs/156]
>
> {code:java}
> 2020-05-15T23:42:29.5307581Z [ERROR]
> testKillYarnSessionClusterEntrypoint(org.apache.flink.yarn.YARNHighAvailabilityITCase)
> Time elapsed: 21.68 s <<< ERROR!
> 2020-05-15T23:42:29.5308406Z java.util.concurrent.ExecutionException:
> 2020-05-15T23:42:29.5308864Z
> org.apache.flink.runtime.rest.util.RestClientException: [Internal server
> error., <Exception on server side:
> 2020-05-15T23:42:29.5309678Z java.util.concurrent.TimeoutException:
> Invocation of public abstract java.util.concurrent.CompletableFuture
> org.apache.flink.runt
> ime.dispatcher.DispatcherGateway.requestJob(org.apache.flink.api.common.JobID,org.apache.flink.api.common.time.Time)
> timed out.
> 2020-05-15T23:42:29.5310322Z at com.sun.proxy.$Proxy33.requestJob(Unknown
> Source)
> 2020-05-15T23:42:29.5311018Z at
> org.apache.flink.runtime.rest.handler.legacy.DefaultExecutionGraphCache.getExecutionGraphInternal(DefaultExecutionGraphCach
> e.java:103)
> 2020-05-15T23:42:29.5311704Z at
> org.apache.flink.runtime.rest.handler.legacy.DefaultExecutionGraphCache.getExecutionGraph(DefaultExecutionGraphCache.java:7
> 1)
> 2020-05-15T23:42:29.5312355Z at
> org.apache.flink.runtime.rest.handler.job.AbstractExecutionGraphHandler.handleRequest(AbstractExecutionGraphHandler.java:75
> )
> 2020-05-15T23:42:29.5312924Z at
> org.apache.flink.runtime.rest.handler.AbstractRestHandler.respondToRequest(AbstractRestHandler.java:73)
> 2020-05-15T23:42:29.5313423Z at
> org.apache.flink.runtime.rest.handler.AbstractHandler.respondAsLeader(AbstractHandler.java:172)
> 2020-05-15T23:42:29.5314497Z at
> org.apache.flink.runtime.rest.handler.LeaderRetrievalHandler.lambda$channelRead0$0(LeaderRetrievalHandler.java:81)
> 2020-05-15T23:42:29.5315083Z at
> java.util.Optional.ifPresent(Optional.java:159)
> 2020-05-15T23:42:29.5315474Z at
> org.apache.flink.util.OptionalConsumer.ifPresent(OptionalConsumer.java:46)
> 2020-05-15T23:42:29.5315979Z at
> org.apache.flink.runtime.rest.handler.LeaderRetrievalHandler.channelRead0(LeaderRetrievalHandler.java:78)
> 2020-05-15T23:42:29.5316520Z at
> org.apache.flink.runtime.rest.handler.LeaderRetrievalHandler.channelRead0(LeaderRetrievalHandler.java:49)
> 2020-05-15T23:42:29.5317092Z at
> org.apache.flink.shaded.netty4.io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:10
> 5)
> 2020-05-15T23:42:29.5317705Z at
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerConte
> xt.java:374)
> 2020-05-15T23:42:29.5318586Z at
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerConte
> xt.java:360)
> 2020-05-15T23:42:29.5319249Z at
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext
> .java:352)
> 2020-05-15T23:42:29.5319729Z at
> org.apache.flink.runtime.rest.handler.router.RouterHandler.routed(RouterHandler.java:110)
> 2020-05-15T23:42:29.5320136Z at
> org.apache.flink.runtime.rest.handler.router.RouterHandler.channelRead0(RouterHandler.java:89)
> 2020-05-15T23:42:29.5320742Z at
> org.apache.flink.runtime.rest.handler.router.RouterHandler.channelRead0(RouterHandler.java:54)
> 2020-05-15T23:42:29.5321195Z at
> org.apache.flink.shaded.netty4.io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:10
> 5)
> 2020-05-15T23:42:29.5321730Z at
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerConte
> xt.java:374)
> 2020-05-15T23:42:29.5322263Z at
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerConte
> xt.java:360)
> 2020-05-15T23:42:29.5322806Z at
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext
> .java:352)
> 2020-05-15T23:42:29.5323335Z at
> org.apache.flink.shaded.netty4.io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
> 2020-05-15T23:42:29.5323849Z at
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerConte
> xt.java:374)
> 2020-05-15T23:42:29.5324390Z at
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerConte
> xt.java:360)
> 2020-05-15T23:42:29.5324910Z at
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext
> .java:352)
> 2020-05-15T23:42:29.5325920Z at
> org.apache.flink.runtime.rest.FileUploadHandler.channelRead0(FileUploadHandler.java:174)
> 2020-05-15T23:42:29.5326353Z at
> org.apache.flink.runtime.rest.FileUploadHandler.channelRead0(FileUploadHandler.java:68)
> 2020-05-15T23:42:29.5326867Z at
> org.apache.flink.shaded.netty4.io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:10
> 5)
> 2020-05-15T23:42:29.5327532Z at
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerConte
> xt.java:374)
> 2020-05-15T23:42:29.5328140Z at
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerConte
> xt.java:360)
> 2020-05-15T23:42:29.5328748Z at
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext
> .java:352)
> 2020-05-15T23:42:29.5329391Z at
> org.apache.flink.shaded.netty4.io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRea
> d(CombinedChannelDuplexHandler.java:438)
> 2020-05-15T23:42:29.5330028Z at
> org.apache.flink.shaded.netty4.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:328)
> 2020-05-15T23:42:29.5330593Z at
> org.apache.flink.shaded.netty4.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:302)
> 2020-05-15T23:42:29.5331149Z at
> org.apache.flink.shaded.netty4.io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:
> 253)
> 2020-05-15T23:42:29.5331750Z at
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerConte
> xt.java:374)
> 2020-05-15T23:42:29.5332346Z at
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerConte
> xt.java:360)
> 2020-05-15T23:42:29.5332956Z at
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext
> .java:352)
> 2020-05-15T23:42:29.5333552Z at
> org.apache.flink.shaded.netty4.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:
> 1421)
> 2020-05-15T23:42:29.5334135Z at
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerConte
> xt.java:374)
> 2020-05-15T23:42:29.5334748Z at
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerConte
> xt.java:360)
> 2020-05-15T23:42:29.5335383Z at
> org.apache.flink.shaded.netty4.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930)
> 2020-05-15T23:42:29.5335952Z at
> org.apache.flink.shaded.netty4.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:1
> 63)
> 2020-05-15T23:42:29.5336485Z at
> org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:697)
> 2020-05-15T23:42:29.5337019Z at
> org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:632)
> 2020-05-15T23:42:29.5337552Z at
> org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:549)
> 2020-05-15T23:42:29.5338023Z at
> org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:511)
> 2020-05-15T23:42:29.5338689Z at
> org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:918)
> 2020-05-15T23:42:29.5339358Z at
> org.apache.flink.shaded.netty4.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
> 2020-05-15T23:42:29.5339869Z at java.lang.Thread.run(Thread.java:748)
> 2020-05-15T23:42:29.5341122Z Caused by: akka.pattern.AskTimeoutException:
> Recipient [Actor[akka://flink/user/rpc/dispatcher_1#-1875884516]] had already
> been t erminated. Message of type
> [org.apache.flink.runtime.rpc.messages.LocalFencedMessage].
> 2020-05-15T23:42:29.5341660Z at
> akka.pattern.AskableActorRef$.recipientTerminatedException(AskSupport.scala:288)
> 2020-05-15T23:42:29.5342018Z at
> akka.pattern.AskableActorRef$.internalAsk$extension(AskSupport.scala:330)
> 2020-05-15T23:42:29.5342321Z at
> akka.pattern.AskSupport$class.ask(AskSupport.scala:81)
> 2020-05-15T23:42:29.5342592Z at
> akka.pattern.package$.ask(package.scala:42)
> 2020-05-15T23:42:29.5342833Z at
> akka.pattern.Patterns$.ask(Patterns.scala:160)
> 2020-05-15T23:42:29.5343180Z at akka.pattern.Patterns.ask(Patterns.scala)
> 2020-05-15T23:42:29.5343485Z at
> org.apache.flink.runtime.rpc.akka.AkkaInvocationHandler.ask(AkkaInvocationHandler.java:361)
> 2020-05-15T23:42:29.5343911Z at
> org.apache.flink.runtime.rpc.akka.FencedAkkaInvocationHandler.ask(FencedAkkaInvocationHandler.java:123)
> 2020-05-15T23:42:29.5344323Z at
> org.apache.flink.runtime.rpc.akka.AkkaInvocationHandler.invokeRpc(AkkaInvocationHandler.java:222)
> 2020-05-15T23:42:29.5344737Z at
> org.apache.flink.runtime.rpc.akka.AkkaInvocationHandler.invoke(AkkaInvocationHandler.java:134)
> 2020-05-15T23:42:29.5345147Z at
> org.apache.flink.runtime.rpc.akka.FencedAkkaInvocationHandler.invoke(FencedAkkaInvocationHandler.java:79)
> 2020-05-15T23:42:29.5345443Z ... 51 more
> 2020-05-15T23:42:29.5345533Z
> 2020-05-15T23:42:29.5345682Z End of exception on server side>]
> 2020-05-15T23:42:29.5345940Z at
> java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
> 2020-05-15T23:42:29.5346289Z at
> java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)
> 2020-05-15T23:42:29.5346680Z at
> org.apache.flink.yarn.YARNHighAvailabilityITCase.lambda$waitUntilJobIsRunning$4(YARNHighAvailabilityITCase.java:312)
> 2020-05-15T23:42:29.5347126Z at
> org.apache.flink.runtime.testutils.CommonTestUtils.waitUntilCondition(CommonTestUtils.java:123)
> 2020-05-15T23:42:29.5347520Z at
> org.apache.flink.runtime.testutils.CommonTestUtils.waitUntilCondition(CommonTestUtils.java:119)
> 2020-05-15T23:42:29.5347950Z at
> org.apache.flink.yarn.YARNHighAvailabilityITCase.waitUntilJobIsRunning(YARNHighAvailabilityITCase.java:310)
> 2020-05-15T23:42:29.5348430Z at
> org.apache.flink.yarn.YARNHighAvailabilityITCase.lambda$testKillYarnSessionClusterEntrypoint$0(YARNHighAvailabilityITCase.j*ava:168)
> 2020-05-15T23:42:29.5348835Z at
> org.apache.flink.yarn.YarnTestBase.runTest(YarnTestBase.java:258)
> 2020-05-15T23:42:29.5349420Z at
> org.apache.flink.yarn.YARNHighAvailabilityITCase.testKillYarnSessionClusterEntrypoint(YARNHighAvailabilityITCase.java:156)
> 2020-05-15T23:42:29.5349984Z at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 2020-05-15T23:42:29.5350385Z at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 2020-05-15T23:42:29.5350772Z at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 2020-05-15T23:42:29.5351125Z at
> java.lang.reflect.Method.invoke(Method.java:498)
> 2020-05-15T23:42:29.5351459Z at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> 2020-05-15T23:42:29.5351868Z at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> 2020-05-15T23:42:29.5352274Z at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> 2020-05-15T23:42:29.5352661Z at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> 2020-05-15T23:42:29.5353219Z at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> 2020-05-15T23:42:29.5353559Z at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> 2020-05-15T23:42:29.5353886Z at
> org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
> 2020-05-15T23:42:29.5354163Z at
> org.junit.rules.RunRules.evaluate(RunRules.java:20)
> 2020-05-15T23:42:29.5354454Z at
> org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> 2020-05-15T23:42:29.5354776Z at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> 2020-05-15T23:42:29.5355144Z at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> 2020-05-15T23:42:29.5355459Z at
> org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> 2020-05-15T23:42:29.5355764Z at
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> 2020-05-15T23:42:29.5356062Z at
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> 2020-05-15T23:42:29.5356382Z at
> org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> 2020-05-15T23:42:29.5356681Z at
> org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> 2020-05-15T23:42:29.5357068Z at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> 2020-05-15T23:42:29.5357432Z at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> 2020-05-15T23:42:29.5357757Z at
> org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
> 2020-05-15T23:42:29.5358088Z at
> org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
> 2020-05-15T23:42:29.5358402Z at
> org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
> 2020-05-15T23:42:29.5358705Z at
> org.junit.rules.RunRules.evaluate(RunRules.java:20)
> 2020-05-15T23:42:29.5358976Z at
> org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> 2020-05-15T23:42:29.5359307Z at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> 2020-05-15T23:42:29.5359674Z at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> 2020-05-15T23:42:29.5360252Z at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> 2020-05-15T23:42:29.5360862Z at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> 2020-05-15T23:42:29.5361296Z at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
> 2020-05-15T23:42:29.5361723Z at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
> 2020-05-15T23:42:29.5362125Z at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
> 2020-05-15T23:42:29.5362503Z at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
> 2020-05-15T23:42:29.5362969Z Suppressed: java.lang.AssertionError: There
> is at least one application on the cluster that is not finished.[App
> application_1 589586092047_0002 is in state RUNNING.]
> 2020-05-15T23:42:29.5363571Z at
> org.junit.Assert.fail(Assert.java:88)
> 2020-05-15T23:42:29.5364060Z at
> org.apache.flink.yarn.YarnTestBase$CleanupYarnApplication.close(YarnTestBase.java:286)
> 2020-05-15T23:42:29.5364448Z at
> org.apache.flink.yarn.YarnTestBase.runTest(YarnTestBase.java:259)
> 2020-05-15T23:42:29.5364745Z ... 36 more
> 2020-05-15T23:42:29.5365232Z Caused by:
> org.apache.flink.runtime.rest.util.RestClientException: [Internal server
> error., <Exception on server side:
> 2020-05-15T23:42:29.5365862Z java.util.concurrent.TimeoutException:
> Invocation of public abstract java.util.concurrent.CompletableFuture
> org.apache.flink.runt
> ime.dispatcher.DispatcherGateway.requestJob(org.apache.flink.api.common.JobID,org.apache.flink.api.common.time.Time)
> timed out.
> 2020-05-15T23:42:29.5366535Z at com.sun.proxy.$Proxy33.requestJob(Unknown
> Source)
> 2020-05-15T23:42:29.5366919Z at
> org.apache.flink.runtime.rest.handler.legacy.DefaultExecutionGraphCache.getExecutionGraphInternal(DefaultExecutionGraphCach
> e.java:103)
> 2020-05-15T23:42:29.5367421Z at
> org.apache.flink.runtime.rest.handler.legacy.DefaultExecutionGraphCache.getExecutionGraph(DefaultExecutionGraphCache.java:7
> 1)
> 2020-05-15T23:42:29.5367925Z at
> org.apache.flink.runtime.rest.handler.job.AbstractExecutionGraphHandler.handleRequest(AbstractExecutionGraphHandler.java:75
> )
> 2020-05-15T23:42:29.5368376Z at
> org.apache.flink.runtime.rest.handler.AbstractRestHandler.respondToRequest(AbstractRestHandler.java:73)
> 2020-05-15T23:42:29.5368902Z at
> org.apache.flink.runtime.rest.handler.AbstractHandler.respondAsLeader(AbstractHandler.java:172)
> 2020-05-15T23:42:29.5369342Z at
> org.apache.flink.runtime.rest.handler.LeaderRetrievalHandler.lambda$channelRead0$0(LeaderRetrievalHandler.java:81)
> 2020-05-15T23:42:29.5369695Z at
> java.util.Optional.ifPresent(Optional.java:159)
> 2020-05-15T23:42:29.5370003Z at
> org.apache.flink.util.OptionalConsumer.ifPresent(OptionalConsumer.java:46)
> 2020-05-15T23:42:29.5370430Z at
> org.apache.flink.runtime.rest.handler.LeaderRetrievalHandler.channelRead0(LeaderRetrievalHandler.java:78)
> 2020-05-15T23:42:29.5370872Z at
> org.apache.flink.runtime.rest.handler.LeaderRetrievalHandler.channelRead0(LeaderRetrievalHandler.java:49)
> 2020-05-15T23:42:29.5371391Z at
> org.apache.flink.shaded.netty4.io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:10
> 5)
> 2020-05-15T23:42:29.5371922Z at
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerConte
> xt.java:374)
> 2020-05-15T23:42:29.5372438Z at
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
> 2020-05-15T23:42:29.5372963Z at
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext
> .java:352)
> 2020-05-15T23:42:29.5373420Z at
> org.apache.flink.runtime.rest.handler.router.RouterHandler.routed(RouterHandler.java:110)
> 2020-05-15T23:42:29.5373804Z at
> org.apache.flink.runtime.rest.handler.router.RouterHandler.channelRead0(RouterHandler.java:89)
> 2020-05-15T23:42:29.5374211Z at
> org.apache.flink.runtime.rest.handler.router.RouterHandler.channelRead0(RouterHandler.java:54)
> 2020-05-15T23:42:29.5374655Z at
> org.apache.flink.shaded.netty4.io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:10
> 5)
> 2020-05-15T23:42:29.5375352Z at
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerConte
> xt.java:374)
> 2020-05-15T23:42:29.5376349Z at
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerConte
> xt.java:360)
> 2020-05-15T23:42:29.5376957Z at
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext
> .java:352)
> 2020-05-15T23:42:29.5377544Z at
> org.apache.flink.shaded.netty4.io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
> 2020-05-15T23:42:29.5378119Z at
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerConte
> xt.java:374)
> 2020-05-15T23:42:29.5378804Z at
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerConte
> xt.java:360)
> 2020-05-15T23:42:29.5379469Z at
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext
> .java:352)
> 2020-05-15T23:42:29.5379997Z at
> org.apache.flink.runtime.rest.FileUploadHandler.channelRead0(FileUploadHandler.java:174)
> 2020-05-15T23:42:29.5381976Z at
> org.apache.flink.runtime.rest.FileUploadHandler.channelRead0(FileUploadHandler.java:68)
> 2020-05-15T23:42:29.5382435Z at
> org.apache.flink.shaded.netty4.io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:10
> 5)
> 2020-05-15T23:42:29.5382951Z at
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerConte
> xt.java:374)
> 2020-05-15T23:42:29.5383467Z at
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerConte
> xt.java:360)
> 2020-05-15T23:42:29.5384003Z at
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext
> .java:352)
> 2020-05-15T23:42:29.5384561Z at
> org.apache.flink.shaded.netty4.io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRea
> d(CombinedChannelDuplexHandler.java:438)
> 2020-05-15T23:42:29.5385116Z at
> org.apache.flink.shaded.netty4.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:328)
> 2020-05-15T23:42:29.5385601Z at
> org.apache.flink.shaded.netty4.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:302)
> 2020-05-15T23:42:29.5386075Z at
> org.apache.flink.shaded.netty4.io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:
> 253)
> 2020-05-15T23:42:29.5386588Z at
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerConte
> xt.java:374)
> 2020-05-15T23:42:29.5387217Z at
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerConte
> xt.java:360)
> 2020-05-15T23:42:29.5387765Z at
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext
> .java:352)
> 2020-05-15T23:42:29.5388263Z at
> org.apache.flink.shaded.netty4.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:
> 1421)
> 2020-05-15T23:42:29.5388785Z at
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerConte
> xt.java:374)
> 2020-05-15T23:42:29.5389313Z at
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerConte
> xt.java:360)
> 2020-05-15T23:42:29.5389799Z at
> org.apache.flink.shaded.netty4.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930)
> 2020-05-15T23:42:29.5390295Z at
> org.apache.flink.shaded.netty4.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:1
> 63)
> 2020-05-15T23:42:29.5390755Z at
> org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:697)
> 2020-05-15T23:42:29.5391213Z at
> org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:632)
> 2020-05-15T23:42:29.5391659Z at
> org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:549)
> 2020-05-15T23:42:29.5392081Z at
> org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:511)
> 2020-05-15T23:42:29.5392527Z at
> org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:918)
> 2020-05-15T23:42:29.5392978Z at
> org.apache.flink.shaded.netty4.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
> 2020-05-15T23:42:29.5393314Z at java.lang.Thread.run(Thread.java:748)
> 2020-05-15T23:42:29.5394130Z Caused by: akka.pattern.AskTimeoutException:
> Recipient [Actor[akka://flink/user/rpc/dispatcher_1#-1875884516]] had already
> been t erminated. Message of type
> [org.apache.flink.runtime.rpc.messages.LocalFencedMessage].
> 2020-05-15T23:42:29.5394945Z at
> akka.pattern.AskableActorRef$.recipientTerminatedException(AskSupport.scala:288)
> 2020-05-15T23:42:29.5395309Z at
> akka.pattern.AskableActorRef$.internalAsk$extension(AskSupport.scala:330)
> 2020-05-15T23:42:29.5395831Z at
> akka.pattern.AskSupport$class.ask(AskSupport.scala:81)
> 2020-05-15T23:42:29.5396125Z at
> akka.pattern.package$.ask(package.scala:42)
> 2020-05-15T23:42:29.5396388Z at
> akka.pattern.Patterns$.ask(Patterns.scala:160)
> 2020-05-15T23:42:29.5396994Z at akka.pattern.Patterns.ask(Patterns.scala)
> 2020-05-15T23:42:29.5397333Z at
> org.apache.flink.runtime.rpc.akka.AkkaInvocationHandler.ask(AkkaInvocationHandler.java:361)
> 2020-05-15T23:42:29.5397823Z at
> org.apache.flink.runtime.rpc.akka.FencedAkkaInvocationHandler.ask(FencedAkkaInvocationHandler.java:123)
> 2020-05-15T23:42:29.5398457Z at
> org.apache.flink.runtime.rpc.akka.AkkaInvocationHandler.invokeRpc(AkkaInvocationHandler.java:222)
> 2020-05-15T23:42:29.5398923Z at
> org.apache.flink.runtime.rpc.akka.AkkaInvocationHandler.invoke(AkkaInvocationHandler.java:134)
> 2020-05-15T23:42:29.5399383Z at
> org.apache.flink.runtime.rpc.akka.FencedAkkaInvocationHandler.invoke(FencedAkkaInvocationHandler.java:79)
> 2020-05-15T23:42:29.5399708Z ... 51 more
> 2020-05-15T23:42:29.5399974Z
> 2020-05-15T23:42:29.5400132Z End of exception on server side>]
> 2020-05-15T23:42:29.5400567Z at
> org.apache.flink.runtime.rest.RestClient.parseResponse(RestClient.java:390)
> 2020-05-15T23:42:29.5400930Z at
> org.apache.flink.runtime.rest.RestClient.lambda$submitRequest$3(RestClient.java:374)
> 2020-05-15T23:42:29.5401283Z at
> java.util.concurrent.CompletableFuture.uniCompose(CompletableFuture.java:966)
> 2020-05-15T23:42:29.5401652Z at
> java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:940)
> 2020-05-15T23:42:29.5402067Z at
> java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456)
> 2020-05-15T23:42:29.5402452Z at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> 2020-05-15T23:42:29.5402800Z at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2020-05-15T23:42:29.5403098Z at java.lang.Thread.run(Thread.java:748)
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
