[ https://issues.apache.org/jira/browse/FLINK-24601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17432418#comment-17432418 ]
Shyam edited comment on FLINK-24601 at 10/21/21, 12:05 PM: ------------------------------------------------------------ The environment setting is very simple. I'm creating an environment like this {code:java} ExecutionEnvironment.createRemoteEnvironment("MY SERVER IP", 8081, "build/libs/dataprocessor-0.0.1-SNAPSHOT.jar");{code} The only repeated exception message which I see in the dashboard log is this : {code:java} 2021-10-21 11:50:22,847 ERROR org.apache.flink.runtime.rest.handler.job.JobSubmitHandler [] - Exception occurred in REST handler: The jobGraphFileName field must not be omitted or be null.{code} The exception I see in the local is {code:java} 2021-10-21 17:19:54.813 INFO 27712 --- [ent-IO-thread-1] o.a.f.c.program.rest.RestClusterClient : Submitting job 'Flink Job' (6c64a08f66c1bec6acc0651fbd732e8a).2021-10-21 17:19:54.813 INFO 27712 --- [ent-IO-thread-1] o.a.f.c.program.rest.RestClusterClient : Submitting job 'Flink Job' (6c64a08f66c1bec6acc0651fbd732e8a).2021-10-21 17:20:00.294 WARN 27712 --- [-netty-thread-1] o.a.f.c.program.rest.RestClusterClient : Attempt to submit job 'Flink Job' (6c64a08f66c1bec6acc0651fbd732e8a) to 'http://xx.xx.xx.xx:8081' has failed. java.util.concurrent.CompletionException: org.apache.flink.runtime.rest.ConnectionClosedException: Channel became inactive. at java.util.concurrent.CompletableFuture.encodeRelay(CompletableFuture.java:326) [na:1.8.0_292] at java.util.concurrent.CompletableFuture.completeRelay(CompletableFuture.java:338) [na:1.8.0_292] at java.util.concurrent.CompletableFuture.uniRelay(CompletableFuture.java:925) [na:1.8.0_292] at java.util.concurrent.CompletableFuture$UniRelay.tryFire(CompletableFuture.java:913) ~[na:1.8.0_292] at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488) [na:1.8.0_292] at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990) [na:1.8.0_292] at org.apache.flink.runtime.rest.RestClient$ClientHandler.channelInactive(RestClient.java:634) ~[flink-runtime-1.14.0.jar:1.14.0] at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262) ~[flink-shaded-netty-4.1.65.Final-14.0.jar:na] at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248) ~[flink-shaded-netty-4.1.65.Final-14.0.jar:na] at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:241) ~[flink-shaded-netty-4.1.65.Final-14.0.jar:na] at org.apache.flink.shaded.netty4.io.netty.channel.ChannelInboundHandlerAdapter.channelInactive(ChannelInboundHandlerAdapter.java:81) ~[flink-shaded-netty-4.1.65.Final-14.0.jar:na] at org.apache.flink.shaded.netty4.io.netty.handler.timeout.IdleStateHandler.channelInactive(IdleStateHandler.java:277) ~[flink-shaded-netty-4.1.65.Final-14.0.jar:na] at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262) ~[flink-shaded-netty-4.1.65.Final-14.0.jar:na] at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248) ~[flink-shaded-netty-4.1.65.Final-14.0.jar:na] at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:241) ~[flink-shaded-netty-4.1.65.Final-14.0.jar:na] at org.apache.flink.shaded.netty4.io.netty.handler.stream.ChunkedWriteHandler.channelInactive(ChunkedWriteHandler.java:137) ~[flink-shaded-netty-4.1.65.Final-14.0.jar:na] at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262) ~[flink-shaded-netty-4.1.65.Final-14.0.jar:na] at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248) ~[flink-shaded-netty-4.1.65.Final-14.0.jar:na] at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:241) ~[flink-shaded-netty-4.1.65.Final-14.0.jar:na] at org.apache.flink.shaded.netty4.io.netty.channel.ChannelInboundHandlerAdapter.channelInactive(ChannelInboundHandlerAdapter.java:81) ~[flink-shaded-netty-4.1.65.Final-14.0.jar:na] at org.apache.flink.shaded.netty4.io.netty.handler.codec.MessageAggregator.channelInactive(MessageAggregator.java:438) ~[flink-shaded-netty-4.1.65.Final-14.0.jar:na] at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262) ~[flink-shaded-netty-4.1.65.Final-14.0.jar:na] at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248) ~[flink-shaded-netty-4.1.65.Final-14.0.jar:na] at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:241) ~[flink-shaded-netty-4.1.65.Final-14.0.jar:na] at org.apache.flink.shaded.netty4.io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelInactive(CombinedChannelDuplexHandler.java:418) ~[flink-shaded-netty-4.1.65.Final-14.0.jar:na] at org.apache.flink.shaded.netty4.io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:389) ~[flink-shaded-netty-4.1.65.Final-14.0.jar:na] at org.apache.flink.shaded.netty4.io.netty.handler.codec.ByteToMessageDecoder.channelInactive(ByteToMessageDecoder.java:354) ~[flink-shaded-netty-4.1.65.Final-14.0.jar:na] at org.apache.flink.shaded.netty4.io.netty.handler.codec.http.HttpClientCodec$Decoder.channelInactive(HttpClientCodec.java:311) ~[flink-shaded-netty-4.1.65.Final-14.0.jar:na] at org.apache.flink.shaded.netty4.io.netty.channel.CombinedChannelDuplexHandler.channelInactive(CombinedChannelDuplexHandler.java:221) ~[flink-shaded-netty-4.1.65.Final-14.0.jar:na] at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262) ~[flink-shaded-netty-4.1.65.Final-14.0.jar:na] at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248) ~[flink-shaded-netty-4.1.65.Final-14.0.jar:na] at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:241) ~[flink-shaded-netty-4.1.65.Final-14.0.jar:na] at org.apache.flink.shaded.netty4.io.netty.channel.DefaultChannelPipeline$HeadContext.channelInactive(DefaultChannelPipeline.java:1405) ~[flink-shaded-netty-4.1.65.Final-14.0.jar:na] at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262) ~[flink-shaded-netty-4.1.65.Final-14.0.jar:na] at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248) ~[flink-shaded-netty-4.1.65.Final-14.0.jar:na] at org.apache.flink.shaded.netty4.io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:901) ~[flink-shaded-netty-4.1.65.Final-14.0.jar:na] at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannel$AbstractUnsafe$8.run(AbstractChannel.java:831) ~[flink-shaded-netty-4.1.65.Final-14.0.jar:na] at org.apache.flink.shaded.netty4.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) ~[flink-shaded-netty-4.1.65.Final-14.0.jar:na] at org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) ~[flink-shaded-netty-4.1.65.Final-14.0.jar:na] at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:497) ~[flink-shaded-netty-4.1.65.Final-14.0.jar:na] at org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[flink-shaded-netty-4.1.65.Final-14.0.jar:na] at org.apache.flink.shaded.netty4.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[flink-shaded-netty-4.1.65.Final-14.0.jar:na] at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_292]Caused by: org.apache.flink.runtime.rest.ConnectionClosedException: Channel became inactive.{code} Seems there is an issue at the rest cluster client which submits the job to the server through REST. I'm calling this API {color:#ffab00}{GET}"/v1/datasets"{color} which gives success JSON response with code 200. was (Author: sshyamprasath): The environment setting is very simple. I'm creating an environment like this {code:java} ExecutionEnvironment.createRemoteEnvironment("MY SERVER IP", 8081, "build/libs/dataprocessor-0.0.1-SNAPSHOT.jar");{code} The only repeated exception message which I see in the dashboard log is this : {code:java} 2021-10-21 11:50:22,847 ERROR org.apache.flink.runtime.rest.handler.job.JobSubmitHandler [] - Exception occurred in REST handler: The jobGraphFileName field must not be omitted or be null.{code} The exception I see in the local is {code:java} 2021-10-21 17:19:54.813 INFO 27712 --- [ent-IO-thread-1] o.a.f.c.program.rest.RestClusterClient : Submitting job 'Flink Streaming Job' (6c64a08f66c1bec6acc0651fbd732e8a).2021-10-21 17:19:54.813 INFO 27712 --- [ent-IO-thread-1] o.a.f.c.program.rest.RestClusterClient : Submitting job 'Flink Streaming Job' (6c64a08f66c1bec6acc0651fbd732e8a).2021-10-21 17:20:00.294 WARN 27712 --- [-netty-thread-1] o.a.f.c.program.rest.RestClusterClient : Attempt to submit job 'Flink Streaming Job' (6c64a08f66c1bec6acc0651fbd732e8a) to 'http://xx.xx.xx.xx:8081' has failed. java.util.concurrent.CompletionException: org.apache.flink.runtime.rest.ConnectionClosedException: Channel became inactive. at java.util.concurrent.CompletableFuture.encodeRelay(CompletableFuture.java:326) [na:1.8.0_292] at java.util.concurrent.CompletableFuture.completeRelay(CompletableFuture.java:338) [na:1.8.0_292] at java.util.concurrent.CompletableFuture.uniRelay(CompletableFuture.java:925) [na:1.8.0_292] at java.util.concurrent.CompletableFuture$UniRelay.tryFire(CompletableFuture.java:913) ~[na:1.8.0_292] at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488) [na:1.8.0_292] at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990) [na:1.8.0_292] at org.apache.flink.runtime.rest.RestClient$ClientHandler.channelInactive(RestClient.java:634) ~[flink-runtime-1.14.0.jar:1.14.0] at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262) ~[flink-shaded-netty-4.1.65.Final-14.0.jar:na] at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248) ~[flink-shaded-netty-4.1.65.Final-14.0.jar:na] at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:241) ~[flink-shaded-netty-4.1.65.Final-14.0.jar:na] at org.apache.flink.shaded.netty4.io.netty.channel.ChannelInboundHandlerAdapter.channelInactive(ChannelInboundHandlerAdapter.java:81) ~[flink-shaded-netty-4.1.65.Final-14.0.jar:na] at org.apache.flink.shaded.netty4.io.netty.handler.timeout.IdleStateHandler.channelInactive(IdleStateHandler.java:277) ~[flink-shaded-netty-4.1.65.Final-14.0.jar:na] at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262) ~[flink-shaded-netty-4.1.65.Final-14.0.jar:na] at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248) ~[flink-shaded-netty-4.1.65.Final-14.0.jar:na] at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:241) ~[flink-shaded-netty-4.1.65.Final-14.0.jar:na] at org.apache.flink.shaded.netty4.io.netty.handler.stream.ChunkedWriteHandler.channelInactive(ChunkedWriteHandler.java:137) ~[flink-shaded-netty-4.1.65.Final-14.0.jar:na] at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262) ~[flink-shaded-netty-4.1.65.Final-14.0.jar:na] at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248) ~[flink-shaded-netty-4.1.65.Final-14.0.jar:na] at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:241) ~[flink-shaded-netty-4.1.65.Final-14.0.jar:na] at org.apache.flink.shaded.netty4.io.netty.channel.ChannelInboundHandlerAdapter.channelInactive(ChannelInboundHandlerAdapter.java:81) ~[flink-shaded-netty-4.1.65.Final-14.0.jar:na] at org.apache.flink.shaded.netty4.io.netty.handler.codec.MessageAggregator.channelInactive(MessageAggregator.java:438) ~[flink-shaded-netty-4.1.65.Final-14.0.jar:na] at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262) ~[flink-shaded-netty-4.1.65.Final-14.0.jar:na] at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248) ~[flink-shaded-netty-4.1.65.Final-14.0.jar:na] at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:241) ~[flink-shaded-netty-4.1.65.Final-14.0.jar:na] at org.apache.flink.shaded.netty4.io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelInactive(CombinedChannelDuplexHandler.java:418) ~[flink-shaded-netty-4.1.65.Final-14.0.jar:na] at org.apache.flink.shaded.netty4.io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:389) ~[flink-shaded-netty-4.1.65.Final-14.0.jar:na] at org.apache.flink.shaded.netty4.io.netty.handler.codec.ByteToMessageDecoder.channelInactive(ByteToMessageDecoder.java:354) ~[flink-shaded-netty-4.1.65.Final-14.0.jar:na] at org.apache.flink.shaded.netty4.io.netty.handler.codec.http.HttpClientCodec$Decoder.channelInactive(HttpClientCodec.java:311) ~[flink-shaded-netty-4.1.65.Final-14.0.jar:na] at org.apache.flink.shaded.netty4.io.netty.channel.CombinedChannelDuplexHandler.channelInactive(CombinedChannelDuplexHandler.java:221) ~[flink-shaded-netty-4.1.65.Final-14.0.jar:na] at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262) ~[flink-shaded-netty-4.1.65.Final-14.0.jar:na] at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248) ~[flink-shaded-netty-4.1.65.Final-14.0.jar:na] at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:241) ~[flink-shaded-netty-4.1.65.Final-14.0.jar:na] at org.apache.flink.shaded.netty4.io.netty.channel.DefaultChannelPipeline$HeadContext.channelInactive(DefaultChannelPipeline.java:1405) ~[flink-shaded-netty-4.1.65.Final-14.0.jar:na] at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262) ~[flink-shaded-netty-4.1.65.Final-14.0.jar:na] at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248) ~[flink-shaded-netty-4.1.65.Final-14.0.jar:na] at org.apache.flink.shaded.netty4.io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:901) ~[flink-shaded-netty-4.1.65.Final-14.0.jar:na] at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannel$AbstractUnsafe$8.run(AbstractChannel.java:831) ~[flink-shaded-netty-4.1.65.Final-14.0.jar:na] at org.apache.flink.shaded.netty4.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) ~[flink-shaded-netty-4.1.65.Final-14.0.jar:na] at org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) ~[flink-shaded-netty-4.1.65.Final-14.0.jar:na] at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:497) ~[flink-shaded-netty-4.1.65.Final-14.0.jar:na] at org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[flink-shaded-netty-4.1.65.Final-14.0.jar:na] at org.apache.flink.shaded.netty4.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[flink-shaded-netty-4.1.65.Final-14.0.jar:na] at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_292]Caused by: org.apache.flink.runtime.rest.ConnectionClosedException: Channel became inactive.{code} Seems there is an issue at the rest cluster client which submits the job to the server through REST. I'm calling this API {color:#ffab00}{GET}"/v1/datasets"{color} which gives success JSON response with code 200. > Error in submitting the job > --------------------------- > > Key: FLINK-24601 > URL: https://issues.apache.org/jira/browse/FLINK-24601 > Project: Flink > Issue Type: Bug > Reporter: Shyam > Priority: Major > > I am new to Flink. I have done Flink 1.14.0 standalone installation in AWS > server and written a simple job in java 1.8. I am new to Flink. I have done > Flink 1.14.0 standalone installation in AWS server and written a simple job > in java 1.8. > > {code:java} > DataSet<String> set = executionEnvironment.fromCollection(text); > DataSet<String> upperCase = set.map(s -> s.toUpperCase()); > upperCase.count(); > executionEnvironment.execute("sample job"); > {code} > > When I run the code, I get the following error in Flink runtime > > {code:java} > 2021-10-19 16:00:51.398 WARN 14358 — [-netty-thread-1] > o.a.f.c.program.rest.RestClusterClient : Attempt to submit job 'Flink Java > Job at Tue Oct 19 16:00:38 IST 2021' (f38d6a987100a98662e10ada551d05b9) to > 'http://xx.xx.xx.xx:8081' has failed. > java.util.concurrent.CompletionException: > org.apache.flink.runtime.rest.ConnectionClosedException: Channel became > inactive. at > java.util.concurrent.CompletableFuture.encodeRelay(CompletableFuture.java:326) > [na:1.8.0_292] at > java.util.concurrent.CompletableFuture.completeRelay(CompletableFuture.java:338) > [na:1.8.0_292] at > java.util.concurrent.CompletableFuture.uniRelay(CompletableFuture.java:925) > [na:1.8.0_292] at > java.util.concurrent.CompletableFuture$UniRelay.tryFire(CompletableFuture.java:913) > ~[na:1.8.0_292] at > java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488) > [na:1.8.0_292] at > java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990) > [na:1.8.0_292] at > org.apache.flink.runtime.rest.RestClient$ClientHandler.channelInactive(RestClient.java:634) > ~[flink-runtime-1.14.0.jar:1.14.0]{code} > > > My job is retrying to submit the job for the multiple times and finally > stopped with another message > {code:java} > org.apache.flink.runtime.rest.util.RestClientException: > [org.apache.flink.runtime.rest.handler.RestHandlerException: The > jobGraphFileName field must not be omitted or be null. at > org.apache.flink.runtime.rest.handler.job.JobSubmitHandler.handleRequest(JobSubmitHandler.java:105){code} > Note: I checked the Flink and java version in my app and server. Everything > is the same (Flink 1.4.0 & java 1.8) -- This message was sent by Atlassian Jira (v8.3.4#803005)