Looking at the lib folder revealed the problem. The lib folder on one of the nodes had libraries for both hadoop 1 and 2. I’m not sure how I ended up with that but it must have happened while I was copying the dependency jars to each node. I removed all the jars and started with a fresh copy and reran the test. It worked this time.
Thank you so much for being patient with me and sorry if I wasted your time. -Ali On 2016-01-04, 3:17 PM, "Robert Metzger" <rmetz...@apache.org> wrote: >Okay, then, my previous statement is false. From the stack trace, it seems >that Flink is using an older Hadoop version. The DFSClient and RPC classes >look different in Hadoop 2.6.0. >Max checked already for some environment variables. Is $CLASSPATH set? Did >you install Hadoop only by downloading the binaries or did you use >something like the ubuntu package manager? (those packages sometimes place >jar files in some lib/ folders) >Did you put something additionally into the lib/ folder of Flink? > >I think when Flink is starting up, its also logging the classpath of the >JVM. Can you post it here? (Maybe you need to start with DEBUG log level) > > >On Mon, Jan 4, 2016 at 8:46 PM, Kashmar, Ali <ali.kash...@emc.com> wrote: > >> Hi Robert, >> >> On the ubuntu host, the port 9000 is up: >> >> ubuntu@ubuntu-171:~$ netstat -tulpn >> .... >> tcp 0 0 10.13.182.171:9000 0.0.0.0:* >>LISTEN >> 8849/java >> .... >> >> And the process using this port is: >> >> ubuntu@ubuntu-171:~$ ps -ef | grep 8849 >> ubuntu 8849 1 0 2015 ? 00:25:00 >> /usr/lib/jvm/java-8-oracle/bin/java -Dproc_namenode -Xmx1000m >> -Djava.net.preferIPv4Stack=true >> -Dhadoop.log.dir=/home/ubuntu/Documents/hadoop-2.6.0/logs >> -Dhadoop.log.file=hadoop.log >> -Dhadoop.home.dir=/home/ubuntu/Documents/hadoop-2.6.0 >> -Dhadoop.id.str=ubuntu -Dhadoop.root.logger=INFO,console >> -Djava.library.path=/home/ubuntu/Documents/hadoop-2.6.0/lib/native >> -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true >> -Djava.net.preferIPv4Stack=true -Djava.net.preferIPv4Stack=true >> -Dhadoop.log.dir=/home/ubuntu/Documents/hadoop-2.6.0/logs >> -Dhadoop.log.file=hadoop-ubuntu-namenode-ubuntu-171.log >> -Dhadoop.home.dir=/home/ubuntu/Documents/hadoop-2.6.0 >> -Dhadoop.id.str=ubuntu -Dhadoop.root.logger=INFO,RFA >> -Djava.library.path=/home/ubuntu/Documents/hadoop-2.6.0/lib/native >> -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true >> -Dhadoop.security.logger=INFO,RFAS -Dhdfs.audit.logger=INFO,NullAppender >> -Dhadoop.security.logger=INFO,RFAS -Dhdfs.audit.logger=INFO,NullAppender >> -Dhadoop.security.logger=INFO,RFAS -Dhdfs.audit.logger=INFO,NullAppender >> -Dhadoop.security.logger=INFO,RFAS >> org.apache.hadoop.hdfs.server.namenode.NameNode >> >> And the hadoop version is: >> >> ubuntu@ubuntu-171:~$ Documents/hadoop-2.6.0/bin/hadoop version >> Hadoop 2.6.0 >> Subversion https://git-wip-us.apache.org/repos/asf/hadoop.git -r >> e3496499ecb8d220fba99dc5ed4c99c8f9e33bb1 >> Compiled by jenkins on 2014-11-13T21:10Z >> Compiled with protoc 2.5.0 >> From source with checksum 18e43357c8f927c0695f1e9522859d6a >> This command was run using >> >>/home/ubuntu/Documents/hadoop-2.6.0/share/hadoop/common/hadoop-common-2.6 >>.0 >> .jar >> >> >> >> -Ali >> >> >> >> On 2016-01-04, 2:20 PM, "Robert Metzger" <rmetz...@apache.org> wrote: >> >> >Your Flink installation has Hadoop 2.6.0 included, on the other >>machine, >> >there is a Hadoop version installed, which is most likely a 1.x or >>even a >> >0.x version. >> >Are you sure that the host "ubuntu-171" has the ip 10.13.182.171 and >>that >> >the hadoop installation in the "/home/ubuntu/Documents/hadoop-2.6.0/" >> >directory is listening on port 9000 ? >> > >> >On Mon, Jan 4, 2016 at 3:03 PM, Kashmar, Ali <ali.kash...@emc.com> >>wrote: >> > >> >> Hi Max, >> >> >> >> Both commands return nothing. Those variables aren’t set. >> >> >> >> The only software I installed on these machines is Flink and Java. >> >> >> >> -Ali >> >> >> >> On 2015-12-28, 6:42 AM, "Maximilian Michels" <m...@apache.org> wrote: >> >> >> >> >Hi Ali, >> >> > >> >> >The warning about the native Hadoop libraries is nothing to worry >> >> >about. The native modules are platform-optimized modules which may >>be >> >> >used to improve performance. They are not necessary for Hadoop to >> >> >function correctly. >> >> > >> >> >The exception message implies that you are using is a very old >>version >> >> >of Hadoop. Do you have other Hadoop versions installed on the same >> >> >machine? We have had people using Flink 0.10.0 with Hadoop 2.6.0 >> >> >without any problems. >> >> > >> >> >On the cluster machines, what is the output of these commands? >> >> > >> >> >echo $HADOOP_CLASSPATH >> >> >echo $HADOOP_CONF_DIR >> >> > >> >> > >> >> >Thanks, >> >> >Max >> >> > >> >> >On Wed, Dec 23, 2015 at 3:53 PM, Kashmar, Ali <ali.kash...@emc.com> >> >> wrote: >> >> >> Hi Max, >> >> >> >> >> >> I have the same output for the Task Manager: >> >> >> >> >> >> 11:25:04,274 INFO >>org.apache.flink.runtime.taskmanager.TaskManager >> >> >> - Hadoop version: 2.6.0 >> >> >> >> >> >> I do get this line at the beginning of both job and task manager >>log >> >> >>files: >> >> >> >> >> >> 11:25:04,100 WARN org.apache.hadoop.util.NativeCodeLoader >> >> >> - Unable to load native-hadoop library for your platform... >> >>using >> >> >> builtin-java classes where applicable >> >> >> >> >> >> Do you think it has anything to do with it? >> >> >> >> >> >> Thanks, >> >> >> Ali >> >> >> >> >> >> On 2015-12-23, 7:30 AM, "Maximilian Michels" <m...@apache.org> >>wrote: >> >> >> >> >> >>>Hi Ali, >> >> >>> >> >> >>>Could you please also post the Hadoop version output of the task >> >> >>>manager log files? It looks like the task managers are running a >> >> >>>different Hadoop version. >> >> >>> >> >> >>>Thanks, >> >> >>>Max >> >> >>> >> >> >>>On Tue, Dec 22, 2015 at 4:28 PM, Kashmar, Ali >><ali.kash...@emc.com> >> >> >>>wrote: >> >> >>>> Hi Robert, >> >> >>>> >> >> >>>> I found the version in the job manager log file: >> >> >>>> >> >> >>>> 17:33:49,636 INFO >>org.apache.flink.runtime.jobmanager.JobManager >> >> >>>> - Hadoop version: 2.6.0 >> >> >>>> >> >> >>>> But the Hadoop installation I have is saying this: >> >> >>>> >> >> >>>> ubuntu@ubuntu-171:~/Documents/hadoop-2.6.0$ bin/hadoop version >> >> >>>> Hadoop 2.6.0 >> >> >>>> Subversion https://git-wip-us.apache.org/repos/asf/hadoop.git -r >> >> >>>> e3496499ecb8d220fba99dc5ed4c99c8f9e33bb1 >> >> >>>> Compiled by jenkins on 2014-11-13T21:10Z >> >> >>>> Compiled with protoc 2.5.0 >> >> >>>> From source with checksum 18e43357c8f927c0695f1e9522859d6a >> >> >>>> This command was run using >> >> >>>> >> >> >> >>>>>>>>/home/ubuntu/Documents/hadoop-2.6.0/share/hadoop/common/hadoop-comm >>>>>>>>on >> >>>>>>-2 >> >> >>>>.6 >> >> >>>>.0 >> >> >>>> .jar >> >> >>>> >> >> >>>> >> >> >>>> So one of them is lying to me? :) >> >> >>>> >> >> >>>> Ali >> >> >>>> >> >> >>>> On 2015-12-22, 10:16 AM, "Robert Metzger" <rmetz...@apache.org> >> >> wrote: >> >> >>>> >> >> >>>>>Hi Ali, >> >> >>>>> >> >> >>>>>the TaskManagers and the JobManager is logging the Hadoop >>version >> >>on >> >> >>>>>startup. >> >> >>>>> >> >> >>>>>On Tue, Dec 22, 2015 at 4:10 PM, Kashmar, Ali >><ali.kash...@emc.com >> > >> >> >>>>>wrote: >> >> >>>>> >> >> >>>>>> Hello, >> >> >>>>>> >> >> >>>>>> I¹m trying to use HDFS as store for Flink checkpoints so I >> >> >>>>>>downloaded >> >> >>>>>>the >> >> >>>>>> Hadoop 2.6.0/Scala 2.10 version of Flink and installed it. I >>also >> >> >>>>>> downloaded Hadoop 2.6.0 separately from the Hadoop website and >> >>set >> >> >>>>>>up >> >> >>>>>>HDFS >> >> >>>>>> on a separate machine. When I start Flink I get the following >> >>error: >> >> >>>>>> >> >> >>>>>> 17:34:13,047 INFO >>org.apache.flink.runtime.jobmanager.JobManager >> >> >>>>>> - Status of job 9ba32a08bc0ec02810bf5d2710842f72 >>(Protocol >> >> >>>>>>Event >> >> >>>>>> Processing) changed to FAILED. >> >> >>>>>> java.lang.Exception: Call to registerInputOutput() of >>invokable >> >> >>>>>>failed >> >> >>>>>> at >> >> >>>>>>org.apache.flink.runtime.taskmanager.Task.run(Task.java:529) >> >> >>>>>> at java.lang.Thread.run(Thread.java:745) >> >> >>>>>> Caused by: java.io.IOException: The given file URI (hdfs:// >> >> >>>>>> 10.13.182.171:9000/user/flink/checkpoints) points to the HDFS >> >> >>>>>>NameNode >> >> >>>>>>at >> >> >>>>>> 10.13.182.171:9000, but the File System could not be >>initialized >> >> >>>>>>with >> >> >>>>>> that address: Server IPC version 9 cannot communicate with >>client >> >> >>>>>>version 4 >> >> >>>>>> at >> >> >>>>>> >> >> >> >>>>>>>>>>org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.initialize(Hado >>>>>>>>>>op >> >>>>>>>>Fi >> >> >>>>>>le >> >> >>>>>>Sy >> >> >>>>>>stem.java:337) >> >> >>>>>> at >> >> >>>>>>org.apache.flink.core.fs.FileSystem.get(FileSystem.java:253) >> >> >>>>>> at >> >> >>>>>> >> >> >> >>>>>>>>>>org.apache.flink.runtime.state.filesystem.FsStateBackend.<init>(F >>>>>>>>>>sS >> >>>>>>>>ta >> >> >>>>>>te >> >> >>>>>>Ba >> >> >>>>>>ckend.java:142) >> >> >>>>>> at >> >> >>>>>> >> >> >> >>>>>>>>>>org.apache.flink.runtime.state.filesystem.FsStateBackend.<init>(F >>>>>>>>>>sS >> >>>>>>>>ta >> >> >>>>>>te >> >> >>>>>>Ba >> >> >>>>>>ckend.java:101) >> >> >>>>>> at >> >> >>>>>> >> >> >> >>>>>>>>>>org.apache.flink.runtime.state.filesystem.FsStateBackendFactory.c >>>>>>>>>>re >> >>>>>>>>at >> >> >>>>>>eF >> >> >>>>>>ro >> >> >>>>>>mConfig(FsStateBackendFactory.java:48) >> >> >>>>>> at >> >> >>>>>> >> >> >> >>>>>>>>>>org.apache.flink.streaming.runtime.tasks.StreamTask.createStateBa >>>>>>>>>>ck >> >>>>>>>>en >> >> >>>>>>d( >> >> >>>>>>St >> >> >>>>>>reamTask.java:517) >> >> >>>>>> at >> >> >>>>>> >> >> >> >>>>>>>>>>org.apache.flink.streaming.runtime.tasks.StreamTask.registerInput >>>>>>>>>>Ou >> >>>>>>>>tp >> >> >>>>>>ut >> >> >>>>>>(S >> >> >>>>>>treamTask.java:171) >> >> >>>>>> at >> >> >>>>>>org.apache.flink.runtime.taskmanager.Task.run(Task.java:526) >> >> >>>>>> ... 1 more >> >> >>>>>> Caused by: >> >> >>>>>> >> >> >> >>>>>>>>>>org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.RPC$V >>>>>>>>>>er >> >>>>>>>>si >> >> >>>>>>on >> >> >>>>>>Mi >> >> >>>>>>smatch): >> >> >>>>>> Server IPC version 9 cannot communicate with client version 4 >> >> >>>>>> at org.apache.hadoop.ipc.Client.call(Client.java:1113) >> >> >>>>>> at >>org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229) >> >> >>>>>> at com.sun.proxy.$Proxy6.getProtocolVersion(Unknown >> >>Source) >> >> >>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native >> >> >>>>>>Method) >> >> >>>>>> at >> >> >>>>>> >> >> >> >>>>>>>>>>sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorI >>>>>>>>>>mp >> >>>>>>>>l. >> >> >>>>>>ja >> >> >>>>>>va >> >> >>>>>>:62) >> >> >>>>>> at >> >> >>>>>> >> >> >> >>>>>>>>>>sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodA >>>>>>>>>>cc >> >>>>>>>>es >> >> >>>>>>so >> >> >>>>>>rI >> >> >>>>>>mpl.java:43) >> >> >>>>>> at java.lang.reflect.Method.invoke(Method.java:497) >> >> >>>>>> at >> >> >>>>>> >> >> >> >>>>>>>>>>org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(Re >>>>>>>>>>tr >> >>>>>>>>yI >> >> >>>>>>nv >> >> >>>>>>oc >> >> >>>>>>ationHandler.java:85) >> >> >>>>>> at >> >> >>>>>> >> >> >> >>>>>>>>>>org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInv >>>>>>>>>>oc >> >>>>>>>>at >> >> >>>>>>io >> >> >>>>>>nH >> >> >>>>>>andler.java:62) >> >> >>>>>> at com.sun.proxy.$Proxy6.getProtocolVersion(Unknown >> >>Source) >> >> >>>>>> at >>org.apache.hadoop.ipc.RPC.checkVersion(RPC.java:422) >> >> >>>>>> at >> >> >>>>>> >> >>org.apache.hadoop.hdfs.DFSClient.createNamenode(DFSClient.java:183) >> >> >>>>>> at >> >> >>>>>>org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:281) >> >> >>>>>> at >> >> >>>>>>org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:245) >> >> >>>>>> at >> >> >>>>>> >> >> >> >>>>>>>>>>org.apache.hadoop.hdfs.DistributedFileSystem.initialize(Distribut >>>>>>>>>>ed >> >>>>>>>>Fi >> >> >>>>>>le >> >> >>>>>>Sy >> >> >>>>>>stem.java:100) >> >> >>>>>> at >> >> >>>>>> >> >> >> >>>>>>>>>>org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.initialize(Hado >>>>>>>>>>op >> >>>>>>>>Fi >> >> >>>>>>le >> >> >>>>>>Sy >> >> >>>>>>stem.java:321) >> >> >>>>>> ... 8 more >> >> >>>>>> >> >> >>>>>> I searched for this error online and it indicates that the >>client >> >> >>>>>>which >> >> >>>>>>is >> >> >>>>>> Flink in this case is at a much lower version. Is there a way >>to >> >> >>>>>>check >> >> >>>>>>the >> >> >>>>>> version of Hadoop packaged with my Flink installation? >> >> >>>>>> >> >> >>>>>> Thanks, >> >> >>>>>> Ali >> >> >>>>>> >> >> >>>> >> >> >> >> >> >> >> >> >>
