big +1 from me as well! On Wed, Jun 24, 2015 at 12:05 PM, Ufuk Celebi <u...@apache.org> wrote:
> I think this is a very good idea and very urgent (because of the issues > you outlined and for the user experience of *not* having to compile your > own version). Big +1. > > On 24 Jun 2015, at 11:45, Robert Metzger <rmetz...@apache.org> wrote: > > > Hi, > > > > I am aware of at least two Flink users which were facing various issues > > with HDFS when using Flink. > > > > *Issues observed:* > > - HDFS client trying to connect to the standby Namenode > > > "org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException): > > Operation category READ is not supported in state standby" > > - java.io.IOException: Bad response ERROR for block > > BP-1335380477-172.22.5.37-1424696786673:blk_1107843111_34301064 from > > datanode 172.22.5.81:50010 > > at > > > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer$ResponseProcessor.run(DFSOutputStream.java:732) > > > > - Caused by: > > > org.apache.hadoop.ipc.RemoteException(java.lang.ArrayIndexOutOfBoundsException): > > 0 > > at > > > org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager.getDatanodeStorageInfos(DatanodeManager.java:478) > > at > > > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.updatePipelineInternal(FSNamesystem.java:6039) > > at > > > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.updatePipeline(FSNamesystem.java:6002) > > > > > > I've added the exceptions to the email so that users facing these issues > > can find a solution for them. > > I suspect that all these issues are caused by the Hadoop 2.2.0 client we > > are packing into the binary releases. > > > > Upgrading the HDFS client to the same version as the HDFS installation > > (say, for example 2.4.1) resolved all issues. > > > > Therefore, I propose to provide Hadoop 2.4.0 and Hadoop 2.6.0 binaries on > > the Flink download page. > > For the 0.9.0 release, I would do another VOTE on providing these two > > binaries. > > > > I've also filed a JIRA to provide a Flink build which doesn't include > > Hadoop at all (relying on the version provided by the user through the > > classpath): https://issues.apache.org/jira/browse/FLINK-2268 > > > > > > Let me know what you think! > > > > Robert > >
