Your Flink installation has Hadoop 2.6.0 included, on the other machine, there is a Hadoop version installed, which is most likely a 1.x or even a 0.x version. Are you sure that the host "ubuntu-171" has the ip 10.13.182.171 and that the hadoop installation in the "/home/ubuntu/Documents/hadoop-2.6.0/" directory is listening on port 9000 ?
On Mon, Jan 4, 2016 at 3:03 PM, Kashmar, Ali <ali.kash...@emc.com> wrote: > Hi Max, > > Both commands return nothing. Those variables aren’t set. > > The only software I installed on these machines is Flink and Java. > > -Ali > > On 2015-12-28, 6:42 AM, "Maximilian Michels" <m...@apache.org> wrote: > > >Hi Ali, > > > >The warning about the native Hadoop libraries is nothing to worry > >about. The native modules are platform-optimized modules which may be > >used to improve performance. They are not necessary for Hadoop to > >function correctly. > > > >The exception message implies that you are using is a very old version > >of Hadoop. Do you have other Hadoop versions installed on the same > >machine? We have had people using Flink 0.10.0 with Hadoop 2.6.0 > >without any problems. > > > >On the cluster machines, what is the output of these commands? > > > >echo $HADOOP_CLASSPATH > >echo $HADOOP_CONF_DIR > > > > > >Thanks, > >Max > > > >On Wed, Dec 23, 2015 at 3:53 PM, Kashmar, Ali <ali.kash...@emc.com> > wrote: > >> Hi Max, > >> > >> I have the same output for the Task Manager: > >> > >> 11:25:04,274 INFO org.apache.flink.runtime.taskmanager.TaskManager > >> - Hadoop version: 2.6.0 > >> > >> I do get this line at the beginning of both job and task manager log > >>files: > >> > >> 11:25:04,100 WARN org.apache.hadoop.util.NativeCodeLoader > >> - Unable to load native-hadoop library for your platform... using > >> builtin-java classes where applicable > >> > >> Do you think it has anything to do with it? > >> > >> Thanks, > >> Ali > >> > >> On 2015-12-23, 7:30 AM, "Maximilian Michels" <m...@apache.org> wrote: > >> > >>>Hi Ali, > >>> > >>>Could you please also post the Hadoop version output of the task > >>>manager log files? It looks like the task managers are running a > >>>different Hadoop version. > >>> > >>>Thanks, > >>>Max > >>> > >>>On Tue, Dec 22, 2015 at 4:28 PM, Kashmar, Ali <ali.kash...@emc.com> > >>>wrote: > >>>> Hi Robert, > >>>> > >>>> I found the version in the job manager log file: > >>>> > >>>> 17:33:49,636 INFO org.apache.flink.runtime.jobmanager.JobManager > >>>> - Hadoop version: 2.6.0 > >>>> > >>>> But the Hadoop installation I have is saying this: > >>>> > >>>> ubuntu@ubuntu-171:~/Documents/hadoop-2.6.0$ bin/hadoop version > >>>> Hadoop 2.6.0 > >>>> Subversion https://git-wip-us.apache.org/repos/asf/hadoop.git -r > >>>> e3496499ecb8d220fba99dc5ed4c99c8f9e33bb1 > >>>> Compiled by jenkins on 2014-11-13T21:10Z > >>>> Compiled with protoc 2.5.0 > >>>> From source with checksum 18e43357c8f927c0695f1e9522859d6a > >>>> This command was run using > >>>> > >>>>/home/ubuntu/Documents/hadoop-2.6.0/share/hadoop/common/hadoop-common-2 > >>>>.6 > >>>>.0 > >>>> .jar > >>>> > >>>> > >>>> So one of them is lying to me? :) > >>>> > >>>> Ali > >>>> > >>>> On 2015-12-22, 10:16 AM, "Robert Metzger" <rmetz...@apache.org> > wrote: > >>>> > >>>>>Hi Ali, > >>>>> > >>>>>the TaskManagers and the JobManager is logging the Hadoop version on > >>>>>startup. > >>>>> > >>>>>On Tue, Dec 22, 2015 at 4:10 PM, Kashmar, Ali <ali.kash...@emc.com> > >>>>>wrote: > >>>>> > >>>>>> Hello, > >>>>>> > >>>>>> I¹m trying to use HDFS as store for Flink checkpoints so I > >>>>>>downloaded > >>>>>>the > >>>>>> Hadoop 2.6.0/Scala 2.10 version of Flink and installed it. I also > >>>>>> downloaded Hadoop 2.6.0 separately from the Hadoop website and set > >>>>>>up > >>>>>>HDFS > >>>>>> on a separate machine. When I start Flink I get the following error: > >>>>>> > >>>>>> 17:34:13,047 INFO org.apache.flink.runtime.jobmanager.JobManager > >>>>>> - Status of job 9ba32a08bc0ec02810bf5d2710842f72 (Protocol > >>>>>>Event > >>>>>> Processing) changed to FAILED. > >>>>>> java.lang.Exception: Call to registerInputOutput() of invokable > >>>>>>failed > >>>>>> at > >>>>>>org.apache.flink.runtime.taskmanager.Task.run(Task.java:529) > >>>>>> at java.lang.Thread.run(Thread.java:745) > >>>>>> Caused by: java.io.IOException: The given file URI (hdfs:// > >>>>>> 10.13.182.171:9000/user/flink/checkpoints) points to the HDFS > >>>>>>NameNode > >>>>>>at > >>>>>> 10.13.182.171:9000, but the File System could not be initialized > >>>>>>with > >>>>>> that address: Server IPC version 9 cannot communicate with client > >>>>>>version 4 > >>>>>> at > >>>>>> > >>>>>>org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.initialize(HadoopFi > >>>>>>le > >>>>>>Sy > >>>>>>stem.java:337) > >>>>>> at > >>>>>>org.apache.flink.core.fs.FileSystem.get(FileSystem.java:253) > >>>>>> at > >>>>>> > >>>>>>org.apache.flink.runtime.state.filesystem.FsStateBackend.<init>(FsSta > >>>>>>te > >>>>>>Ba > >>>>>>ckend.java:142) > >>>>>> at > >>>>>> > >>>>>>org.apache.flink.runtime.state.filesystem.FsStateBackend.<init>(FsSta > >>>>>>te > >>>>>>Ba > >>>>>>ckend.java:101) > >>>>>> at > >>>>>> > >>>>>>org.apache.flink.runtime.state.filesystem.FsStateBackendFactory.creat > >>>>>>eF > >>>>>>ro > >>>>>>mConfig(FsStateBackendFactory.java:48) > >>>>>> at > >>>>>> > >>>>>>org.apache.flink.streaming.runtime.tasks.StreamTask.createStateBacken > >>>>>>d( > >>>>>>St > >>>>>>reamTask.java:517) > >>>>>> at > >>>>>> > >>>>>>org.apache.flink.streaming.runtime.tasks.StreamTask.registerInputOutp > >>>>>>ut > >>>>>>(S > >>>>>>treamTask.java:171) > >>>>>> at > >>>>>>org.apache.flink.runtime.taskmanager.Task.run(Task.java:526) > >>>>>> ... 1 more > >>>>>> Caused by: > >>>>>> > >>>>>>org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.RPC$Versi > >>>>>>on > >>>>>>Mi > >>>>>>smatch): > >>>>>> Server IPC version 9 cannot communicate with client version 4 > >>>>>> at org.apache.hadoop.ipc.Client.call(Client.java:1113) > >>>>>> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229) > >>>>>> at com.sun.proxy.$Proxy6.getProtocolVersion(Unknown Source) > >>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native > >>>>>>Method) > >>>>>> at > >>>>>> > >>>>>>sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl. > >>>>>>ja > >>>>>>va > >>>>>>:62) > >>>>>> at > >>>>>> > >>>>>>sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces > >>>>>>so > >>>>>>rI > >>>>>>mpl.java:43) > >>>>>> at java.lang.reflect.Method.invoke(Method.java:497) > >>>>>> at > >>>>>> > >>>>>>org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryI > >>>>>>nv > >>>>>>oc > >>>>>>ationHandler.java:85) > >>>>>> at > >>>>>> > >>>>>>org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocat > >>>>>>io > >>>>>>nH > >>>>>>andler.java:62) > >>>>>> at com.sun.proxy.$Proxy6.getProtocolVersion(Unknown Source) > >>>>>> at org.apache.hadoop.ipc.RPC.checkVersion(RPC.java:422) > >>>>>> at > >>>>>> org.apache.hadoop.hdfs.DFSClient.createNamenode(DFSClient.java:183) > >>>>>> at > >>>>>>org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:281) > >>>>>> at > >>>>>>org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:245) > >>>>>> at > >>>>>> > >>>>>>org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFi > >>>>>>le > >>>>>>Sy > >>>>>>stem.java:100) > >>>>>> at > >>>>>> > >>>>>>org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.initialize(HadoopFi > >>>>>>le > >>>>>>Sy > >>>>>>stem.java:321) > >>>>>> ... 8 more > >>>>>> > >>>>>> I searched for this error online and it indicates that the client > >>>>>>which > >>>>>>is > >>>>>> Flink in this case is at a much lower version. Is there a way to > >>>>>>check > >>>>>>the > >>>>>> version of Hadoop packaged with my Flink installation? > >>>>>> > >>>>>> Thanks, > >>>>>> Ali > >>>>>> > >>>> > >> > >
