What was the log message shown on DEBUG level? Maybe it makes sense to promote it to INFO. ;)
I guess there is no easy way to verify the version, right Max or Robert? On 14 November 2016 at 10:45:52, Gyula Fóra (gyula.f...@gmail.com) wrote: > Hi, > > The main problem was that whatever was going wrong was not apparent in the > Flink Application master runner but it was only shown in the YarnClient > debug log. > > If you run with the default INFO log level all you see that the Yarn client > is trying to fail over again and again as if something was wrong with the > resource manager. Setting it to debug actually shows the error. > > Also it would be great if there was a way to verify YARN versions and > incompatibility, not sure if this is possible easily. > > Gyula > > Ufuk Celebi ezt írta (időpont: 2016. nov. 14., H, 9:42): > > > Good to know that you solved this. :) Do you think there is something we > > can do to help users noticing this situation faster? > > > > – Ufuk > > > > On 13 November 2016 at 00:23:21, Gyula Fóra (gyula.f...@gmail.com) wrote: > > > Hi, > > > > > > What happened is that I compiled Flink with the wrong hadoop version... > > > > > > Sorry :) > > > Gyula > > > > > > Gyula Fóra ezt írta (időpont: 2016. nov. 12., Szo, > > > 13:11): > > > > > > > Hi, > > > > > > > > I am running into some strange issues on yarn with Flink 1.1.3 & 4. For > > > > some reason I started getting this error (see under text.) > > > > The job manager starts and the application is in Accepted state but > > cannot > > > > seem to be able to communicate with the scheduler. (0.0.0.0:8030 seems > > > > strange) > > > > > > > > I didn't change anything on the yarn cluster and this seemed to work > > > > previously (but I just cant get it to work now). The yarn-site.xml > > contains > > > > the proper rm addresses. > > > > > > > > Anybody has any ideas where to go from here? > > > > > > > > Cheers, > > > > Gyula > > > > > > > > JM log: > > > > > > > > 2016-11-12 11:56:06,894 DEBUG org.apache.hadoop.ipc.Client - The ping > > interval > > > is 60000 ms. > > > > 2016-11-12 11:56:06,894 DEBUG org.apache.hadoop.ipc.Client - > > Connecting to /0.0.0.0:8030 > > > > 2016-11-12 11:56:06,899 DEBUG org.apache.hadoop.ipc.Client - closing > > ipc connection > > > to 0.0.0.0/0.0.0.0:8030: Connection refused > > > > > > > > java.net.ConnectException: Call From > > splat24.sto.midasplayer.com/172.25.86.166 > > > to 0.0.0.0:8030 failed on connection exception: > > java.net.ConnectException: Connection > > > refused; For more details see: > > http://wiki.apache.org/hadoop/ConnectionRefused > > > > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native > > Method) > > > > at > > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > > > > > > at > > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > > > > > > at java.lang.reflect.Constructor.newInstance(Constructor.java:422) > > > > at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:783) > > > > at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:730) > > > > at org.apache.hadoop.ipc.Client.call(Client.java:1410) > > > > at org.apache.hadoop.ipc.Client.call(Client.java:1359) > > > > at > > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206) > > > > > > at com.sun.proxy.$Proxy8.registerApplicationMaster(Unknown Source) > > > > at > > org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.registerApplicationMaster(ApplicationMasterProtocolPBClientImpl.java:106) > > > > > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > > > at > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > > > > > > at > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > > > > > at java.lang.reflect.Method.invoke(Method.java:497) > > > > at > > org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186) > > > > > > at > > org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) > > > > > > at com.sun.proxy.$Proxy9.registerApplicationMaster(Unknown Source) > > > > at > > org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl.registerApplicationMaster(AMRMClientImpl.java:196) > > > > > > at > > org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl.registerApplicationMaster(AMRMClientAsyncImpl.java:138) > > > > > > at > > org.apache.flink.yarn.YarnFlinkResourceManager.initialize(YarnFlinkResourceManager.java:259) > > > > > > at > > org.apache.flink.runtime.clusterframework.FlinkResourceManager.preStart(FlinkResourceManager.java:185) > > > > > > at akka.actor.Actor$class.aroundPreStart(Actor.scala:470) > > > > at akka.actor.UntypedActor.aroundPreStart(UntypedActor.scala:97) > > > > at akka.actor.ActorCell.create(ActorCell.scala:580) > > > > at akka.actor.ActorCell.invokeAll$1(ActorCell.scala:456) > > > > at akka.actor.ActorCell.systemInvoke(ActorCell.scala:478) > > > > > > > > > > > > Client: > > > > > > > > 2016-11-12 12:31:31,080 INFO > > org.apache.flink.yarn.cli.FlinkYarnSessionCli > > > - No path for the flink jar passed. Using the location of class > > org.apache.flink.yarn.YarnClusterDescriptor > > > to locate the jar > > > > 2016-11-12 12:31:31,080 INFO > > org.apache.flink.yarn.cli.FlinkYarnSessionCli > > > - No path for the flink jar passed. Using the location of class > > org.apache.flink.yarn.YarnClusterDescriptor > > > to locate the jar > > > > 2016-11-12 12:31:31,101 INFO > > org.apache.flink.yarn.YarnClusterDescriptor - > > > Using values: > > > > 2016-11-12 12:31:31,101 INFO > > org.apache.flink.yarn.YarnClusterDescriptor - > > > TaskManager count = 1 > > > > 2016-11-12 12:31:31,101 INFO > > org.apache.flink.yarn.YarnClusterDescriptor - > > > JobManager memory = 1024 > > > > 2016-11-12 12:31:31,102 INFO > > org.apache.flink.yarn.YarnClusterDescriptor - > > > TaskManager memory = 11000 > > > > 2016-11-12 12:31:31,119 INFO org.apache.hadoop.yarn.client.RMProxy - > > Connecting > > > to ResourceManager at /0.0.0.0:8032 > > > > 2016-11-12 12:31:31,394 WARN > > org.apache.flink.yarn.YarnClusterDescriptor - > > > The file system scheme is 'file'. This indicates that the specified > > Hadoop configuration > > > path is wrong and the system is using the default Hadoop configuration > > values.The Flink > > > YARN client needs to store its files in a distributed file system > > > > 2016-11-12 12:31:31,457 INFO org.apache.flink.yarn.Utils - Copying > > from file:/fjord/sites/flink-1.1.3/conf/log4j.properties > > > to > > file:/fjord/splat/flink/yarn/.flink/application_1478896050022_0013/log4j.properties > > > > > > 2016-11-12 12:31:42,321 INFO org.apache.flink.yarn.Utils - Copying > > from file:/fjord/sites/flink-1.1.3/lib > > > to file:/fjord/splat/flink/yarn/.flink/application_1478896050022_0013/lib > > > > > > > 2016-11-12 12:32:18,457 INFO org.apache.flink.yarn.Utils - Copying > > from file:/fjord/sites/rbea/rbea-on-flink-1.0-SNAPSHOT.jar > > > to > > file:/fjord/splat/flink/yarn/.flink/application_1478896050022_0013/rbea-on-flink-1.0-SNAPSHOT.jar > > > > > > 2016-11-12 12:32:39,725 INFO org.apache.flink.yarn.Utils - Copying > > from file:/fjord/sites/flink-1.1.3/lib/flink-dist_2.10-1.1.4.jar > > > to > > file:/fjord/splat/flink/yarn/.flink/application_1478896050022_0013/flink-dist_2.10-1.1.4.jar > > > > > > 2016-11-12 12:32:58,154 INFO org.apache.flink.yarn.Utils - Copying > > from /fjord/sites/flink-1.1.3/conf/flink-conf.yaml > > > to > > file:/fjord/splat/flink/yarn/.flink/application_1478896050022_0013/flink-conf.yaml > > > > > > 2016-11-12 12:33:02,218 INFO > > org.apache.flink.yarn.YarnClusterDescriptor - > > > Submitting application master application_1478896050022_0013 > > > > 2016-11-12 12:33:02,256 INFO > > org.apache.hadoop.yarn.client.api.impl.YarnClientImpl > > > - Submitted application application_1478896050022_0013 > > > > 2016-11-12 12:33:02,257 INFO > > org.apache.flink.yarn.YarnClusterDescriptor - > > > Waiting for the cluster to be allocated > > > > 2016-11-12 12:33:02,259 INFO > > org.apache.flink.yarn.YarnClusterDescriptor - > > > Deploying cluster, current state ACCEPTED > > > > 2016-11-12 12:34:02,485 INFO > > org.apache.flink.yarn.YarnClusterDescriptor - > > > Deployment took more than 60 seconds. Please check if the requested > > resources are available > > > in the YARN cluster > > > > > > > > > > > > > > > >
