Hi Robert,
On the ubuntu host, the port 9000 is up:
ubuntu@ubuntu-171:~$ netstat -tulpn
....
tcp 0 0 10.13.182.171:9000 0.0.0.0:* LISTEN
8849/java
....
And the process using this port is:
ubuntu@ubuntu-171:~$ ps -ef | grep 8849
ubuntu 8849 1 0 2015 ? 00:25:00
/usr/lib/jvm/java-8-oracle/bin/java -Dproc_namenode -Xmx1000m
-Djava.net.preferIPv4Stack=true
-Dhadoop.log.dir=/home/ubuntu/Documents/hadoop-2.6.0/logs
-Dhadoop.log.file=hadoop.log
-Dhadoop.home.dir=/home/ubuntu/Documents/hadoop-2.6.0
-Dhadoop.id.str=ubuntu -Dhadoop.root.logger=INFO,console
-Djava.library.path=/home/ubuntu/Documents/hadoop-2.6.0/lib/native
-Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true
-Djava.net.preferIPv4Stack=true -Djava.net.preferIPv4Stack=true
-Dhadoop.log.dir=/home/ubuntu/Documents/hadoop-2.6.0/logs
-Dhadoop.log.file=hadoop-ubuntu-namenode-ubuntu-171.log
-Dhadoop.home.dir=/home/ubuntu/Documents/hadoop-2.6.0
-Dhadoop.id.str=ubuntu -Dhadoop.root.logger=INFO,RFA
-Djava.library.path=/home/ubuntu/Documents/hadoop-2.6.0/lib/native
-Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true
-Dhadoop.security.logger=INFO,RFAS -Dhdfs.audit.logger=INFO,NullAppender
-Dhadoop.security.logger=INFO,RFAS -Dhdfs.audit.logger=INFO,NullAppender
-Dhadoop.security.logger=INFO,RFAS -Dhdfs.audit.logger=INFO,NullAppender
-Dhadoop.security.logger=INFO,RFAS
org.apache.hadoop.hdfs.server.namenode.NameNode
And the hadoop version is:
ubuntu@ubuntu-171:~$ Documents/hadoop-2.6.0/bin/hadoop version
Hadoop 2.6.0
Subversion https://git-wip-us.apache.org/repos/asf/hadoop.git -r
e3496499ecb8d220fba99dc5ed4c99c8f9e33bb1
Compiled by jenkins on 2014-11-13T21:10Z
Compiled with protoc 2.5.0
From source with checksum 18e43357c8f927c0695f1e9522859d6a
This command was run using
/home/ubuntu/Documents/hadoop-2.6.0/share/hadoop/common/hadoop-common-2.6.0
.jar
-Ali
On 2016-01-04, 2:20 PM, "Robert Metzger" <rmetz...@apache.org> wrote:
>Your Flink installation has Hadoop 2.6.0 included, on the other machine,
>there is a Hadoop version installed, which is most likely a 1.x or even a
>0.x version.
>Are you sure that the host "ubuntu-171" has the ip 10.13.182.171 and that
>the hadoop installation in the "/home/ubuntu/Documents/hadoop-2.6.0/"
>directory is listening on port 9000 ?
>
>On Mon, Jan 4, 2016 at 3:03 PM, Kashmar, Ali <ali.kash...@emc.com> wrote:
>
>> Hi Max,
>>
>> Both commands return nothing. Those variables aren’t set.
>>
>> The only software I installed on these machines is Flink and Java.
>>
>> -Ali
>>
>> On 2015-12-28, 6:42 AM, "Maximilian Michels" <m...@apache.org> wrote:
>>
>> >Hi Ali,
>> >
>> >The warning about the native Hadoop libraries is nothing to worry
>> >about. The native modules are platform-optimized modules which may be
>> >used to improve performance. They are not necessary for Hadoop to
>> >function correctly.
>> >
>> >The exception message implies that you are using is a very old version
>> >of Hadoop. Do you have other Hadoop versions installed on the same
>> >machine? We have had people using Flink 0.10.0 with Hadoop 2.6.0
>> >without any problems.
>> >
>> >On the cluster machines, what is the output of these commands?
>> >
>> >echo $HADOOP_CLASSPATH
>> >echo $HADOOP_CONF_DIR
>> >
>> >
>> >Thanks,
>> >Max
>> >
>> >On Wed, Dec 23, 2015 at 3:53 PM, Kashmar, Ali <ali.kash...@emc.com>
>> wrote:
>> >> Hi Max,
>> >>
>> >> I have the same output for the Task Manager:
>> >>
>> >> 11:25:04,274 INFO org.apache.flink.runtime.taskmanager.TaskManager
>> >> - Hadoop version: 2.6.0
>> >>
>> >> I do get this line at the beginning of both job and task manager log
>> >>files:
>> >>
>> >> 11:25:04,100 WARN org.apache.hadoop.util.NativeCodeLoader
>> >> - Unable to load native-hadoop library for your platform...
>>using
>> >> builtin-java classes where applicable
>> >>
>> >> Do you think it has anything to do with it?
>> >>
>> >> Thanks,
>> >> Ali
>> >>
>> >> On 2015-12-23, 7:30 AM, "Maximilian Michels" <m...@apache.org> wrote:
>> >>
>> >>>Hi Ali,
>> >>>
>> >>>Could you please also post the Hadoop version output of the task
>> >>>manager log files? It looks like the task managers are running a
>> >>>different Hadoop version.
>> >>>
>> >>>Thanks,
>> >>>Max
>> >>>
>> >>>On Tue, Dec 22, 2015 at 4:28 PM, Kashmar, Ali <ali.kash...@emc.com>
>> >>>wrote:
>> >>>> Hi Robert,
>> >>>>
>> >>>> I found the version in the job manager log file:
>> >>>>
>> >>>> 17:33:49,636 INFO org.apache.flink.runtime.jobmanager.JobManager
>> >>>> - Hadoop version: 2.6.0
>> >>>>
>> >>>> But the Hadoop installation I have is saying this:
>> >>>>
>> >>>> ubuntu@ubuntu-171:~/Documents/hadoop-2.6.0$ bin/hadoop version
>> >>>> Hadoop 2.6.0
>> >>>> Subversion https://git-wip-us.apache.org/repos/asf/hadoop.git -r
>> >>>> e3496499ecb8d220fba99dc5ed4c99c8f9e33bb1
>> >>>> Compiled by jenkins on 2014-11-13T21:10Z
>> >>>> Compiled with protoc 2.5.0
>> >>>> From source with checksum 18e43357c8f927c0695f1e9522859d6a
>> >>>> This command was run using
>> >>>>
>>
>>>>>>/home/ubuntu/Documents/hadoop-2.6.0/share/hadoop/common/hadoop-common
>>>>>>-2
>> >>>>.6
>> >>>>.0
>> >>>> .jar
>> >>>>
>> >>>>
>> >>>> So one of them is lying to me? :)
>> >>>>
>> >>>> Ali
>> >>>>
>> >>>> On 2015-12-22, 10:16 AM, "Robert Metzger" <rmetz...@apache.org>
>> wrote:
>> >>>>
>> >>>>>Hi Ali,
>> >>>>>
>> >>>>>the TaskManagers and the JobManager is logging the Hadoop version
>>on
>> >>>>>startup.
>> >>>>>
>> >>>>>On Tue, Dec 22, 2015 at 4:10 PM, Kashmar, Ali <ali.kash...@emc.com>
>> >>>>>wrote:
>> >>>>>
>> >>>>>> Hello,
>> >>>>>>
>> >>>>>> I¹m trying to use HDFS as store for Flink checkpoints so I
>> >>>>>>downloaded
>> >>>>>>the
>> >>>>>> Hadoop 2.6.0/Scala 2.10 version of Flink and installed it. I also
>> >>>>>> downloaded Hadoop 2.6.0 separately from the Hadoop website and
>>set
>> >>>>>>up
>> >>>>>>HDFS
>> >>>>>> on a separate machine. When I start Flink I get the following
>>error:
>> >>>>>>
>> >>>>>> 17:34:13,047 INFO org.apache.flink.runtime.jobmanager.JobManager
>> >>>>>> - Status of job 9ba32a08bc0ec02810bf5d2710842f72 (Protocol
>> >>>>>>Event
>> >>>>>> Processing) changed to FAILED.
>> >>>>>> java.lang.Exception: Call to registerInputOutput() of invokable
>> >>>>>>failed
>> >>>>>> at
>> >>>>>>org.apache.flink.runtime.taskmanager.Task.run(Task.java:529)
>> >>>>>> at java.lang.Thread.run(Thread.java:745)
>> >>>>>> Caused by: java.io.IOException: The given file URI (hdfs://
>> >>>>>> 10.13.182.171:9000/user/flink/checkpoints) points to the HDFS
>> >>>>>>NameNode
>> >>>>>>at
>> >>>>>> 10.13.182.171:9000, but the File System could not be initialized
>> >>>>>>with
>> >>>>>> that address: Server IPC version 9 cannot communicate with client
>> >>>>>>version 4
>> >>>>>> at
>> >>>>>>
>>
>>>>>>>>org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.initialize(Hadoop
>>>>>>>>Fi
>> >>>>>>le
>> >>>>>>Sy
>> >>>>>>stem.java:337)
>> >>>>>> at
>> >>>>>>org.apache.flink.core.fs.FileSystem.get(FileSystem.java:253)
>> >>>>>> at
>> >>>>>>
>>
>>>>>>>>org.apache.flink.runtime.state.filesystem.FsStateBackend.<init>(FsS
>>>>>>>>ta
>> >>>>>>te
>> >>>>>>Ba
>> >>>>>>ckend.java:142)
>> >>>>>> at
>> >>>>>>
>>
>>>>>>>>org.apache.flink.runtime.state.filesystem.FsStateBackend.<init>(FsS
>>>>>>>>ta
>> >>>>>>te
>> >>>>>>Ba
>> >>>>>>ckend.java:101)
>> >>>>>> at
>> >>>>>>
>>
>>>>>>>>org.apache.flink.runtime.state.filesystem.FsStateBackendFactory.cre
>>>>>>>>at
>> >>>>>>eF
>> >>>>>>ro
>> >>>>>>mConfig(FsStateBackendFactory.java:48)
>> >>>>>> at
>> >>>>>>
>>
>>>>>>>>org.apache.flink.streaming.runtime.tasks.StreamTask.createStateBack
>>>>>>>>en
>> >>>>>>d(
>> >>>>>>St
>> >>>>>>reamTask.java:517)
>> >>>>>> at
>> >>>>>>
>>
>>>>>>>>org.apache.flink.streaming.runtime.tasks.StreamTask.registerInputOu
>>>>>>>>tp
>> >>>>>>ut
>> >>>>>>(S
>> >>>>>>treamTask.java:171)
>> >>>>>> at
>> >>>>>>org.apache.flink.runtime.taskmanager.Task.run(Task.java:526)
>> >>>>>> ... 1 more
>> >>>>>> Caused by:
>> >>>>>>
>>
>>>>>>>>org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.RPC$Ver
>>>>>>>>si
>> >>>>>>on
>> >>>>>>Mi
>> >>>>>>smatch):
>> >>>>>> Server IPC version 9 cannot communicate with client version 4
>> >>>>>> at org.apache.hadoop.ipc.Client.call(Client.java:1113)
>> >>>>>> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229)
>> >>>>>> at com.sun.proxy.$Proxy6.getProtocolVersion(Unknown
>>Source)
>> >>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>> >>>>>>Method)
>> >>>>>> at
>> >>>>>>
>>
>>>>>>>>sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImp
>>>>>>>>l.
>> >>>>>>ja
>> >>>>>>va
>> >>>>>>:62)
>> >>>>>> at
>> >>>>>>
>>
>>>>>>>>sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcc
>>>>>>>>es
>> >>>>>>so
>> >>>>>>rI
>> >>>>>>mpl.java:43)
>> >>>>>> at java.lang.reflect.Method.invoke(Method.java:497)
>> >>>>>> at
>> >>>>>>
>>
>>>>>>>>org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(Retr
>>>>>>>>yI
>> >>>>>>nv
>> >>>>>>oc
>> >>>>>>ationHandler.java:85)
>> >>>>>> at
>> >>>>>>
>>
>>>>>>>>org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvoc
>>>>>>>>at
>> >>>>>>io
>> >>>>>>nH
>> >>>>>>andler.java:62)
>> >>>>>> at com.sun.proxy.$Proxy6.getProtocolVersion(Unknown
>>Source)
>> >>>>>> at org.apache.hadoop.ipc.RPC.checkVersion(RPC.java:422)
>> >>>>>> at
>> >>>>>>
>>org.apache.hadoop.hdfs.DFSClient.createNamenode(DFSClient.java:183)
>> >>>>>> at
>> >>>>>>org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:281)
>> >>>>>> at
>> >>>>>>org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:245)
>> >>>>>> at
>> >>>>>>
>>
>>>>>>>>org.apache.hadoop.hdfs.DistributedFileSystem.initialize(Distributed
>>>>>>>>Fi
>> >>>>>>le
>> >>>>>>Sy
>> >>>>>>stem.java:100)
>> >>>>>> at
>> >>>>>>
>>
>>>>>>>>org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.initialize(Hadoop
>>>>>>>>Fi
>> >>>>>>le
>> >>>>>>Sy
>> >>>>>>stem.java:321)
>> >>>>>> ... 8 more
>> >>>>>>
>> >>>>>> I searched for this error online and it indicates that the client
>> >>>>>>which
>> >>>>>>is
>> >>>>>> Flink in this case is at a much lower version. Is there a way to
>> >>>>>>check
>> >>>>>>the
>> >>>>>> version of Hadoop packaged with my Flink installation?
>> >>>>>>
>> >>>>>> Thanks,
>> >>>>>> Ali
>> >>>>>>
>> >>>>
>> >>
>>
>>
