Hi Colin, I have a requirement to be able to run all the Hadoop functionality that I build from a variety of platforms. This was the original motivation for wanting to use libhdfs. I followed your suggestion and looked into using webhdfs and it's looking promising. Thanks for that. However I also need to be able to launch Map/Reduce jobs from any platform. In particular from Windows. I looked into this by hacking the bin/hadoop script to extract the required class path and various arguments so that I could launch a Map/Reduce job just by invoking java with the correct arguments. However I ran into HADOOP-7682. I can see that there is a workaround here https://github.com/congainc/patch-hadoop_7682-1.0.x-win but it suggests that this is not really appropriate for deployment. I suspect that I can get it to work reliably by using cygwin and making loads of modifications but that all seems rather a large effort, error-prone and difficult to maintain.
Given that I plan to have a relatively small repertoire of Map/Reduce jobs that I need to launch, I'm tempted to have all the jars pre-packed on the Name-Node and have the ability to run them there. Then have a daemon running so that I can use any appropriate ad-hoc RPC mechanism from Windows to launch them. Am I missing something? Is there a way to launch Map/Reduce jobs in a platform neutral way, which runs out of the box, on Windows? Again, any suggestions welcome. Peter Marron > -----Original Message----- > From: Peter Marron > [mailto:Peter.Marron@trilliumsoftware. > com] > Sent: 26 October 2012 00:53 > To: common-dev@hadoop.apache.org > Subject: RE: libhdfs on windows > > Hi Colin, > > OK, I didn't know there was a hdfs-dev. > I'm happy to ask there. > (However there's a lot of mail on > dev@hadoop and user@hadoop as well > as user@hive and it's a bit of a > commitment to track them all.) As for > webhdfs, I did think about that, and in > some ways it's a beautiful solution as it > gives me a platform- and language- > neutral access mechanism. I was just a > little worried about the HTTP overhead if > I am reading a single record at a time. > Also I will need some way to launch my > Map/Reduce jobs as well. So I'll probably > end up using the C++/JNI/Java route to > do that anyway. Unless there's a better > way? > Is there a web Map/Reduce interface? > > Many thanks, > > Z > > > -----Original Message----- > > From: rarecac...@gmail.com > > [mailto:rarecac...@gmail.com] On > > Behalf Of Colin McCabe > > Sent: 25 October 2012 18:24 > > To: common-dev@hadoop.apache.org > > Subject: Re: libhdfs on windows > > > > Hi Peter, > > > > This might be a good question for hdfs- > dev? > > > > As Harsh pointed out below, HDFS-573 > > was never committed. I don't even see > a patch attached, although > > there is some discussion. > > > > In the mean time, might I suggest using > the webhdfs interface on > > Windows? > > webhdfs was intended as a stable REST > > interface that can be accessed from > any platform. > > > > cheers, > > Colin > > > > > > On Thu, Oct 25, 2012 at 7:19 AM, Peter > Marron > > <peter.mar...@trilliumsoftware.com> > > wrote: > > > Hi, > > > > > > I've been looking at using libhdfs and > I > > would like to use it on windows. > > > I have found HDFS-573 and the > > information on this page: > > > > > > http://issues.apache.org/jira/browse/HD > > FS-573?page=com.atlassian.jira. > > > plugin.system.issuetabpanels:all- > > tabpanel > > > which suggests that quite a lot of > work > > was done on this way back in 2009. > > > So is there some source from this > > effort retained somewhere? If so, > > where? > > > Or do I have to start from scratch? > > Apologies if this has already been asked > recently. > > > > > > Any help appreciated. > > > > > > Peter Marron > >
