Re: Data Locality with WebHDFS

2014-03-19 Thread RJ Nowling
Hi Steve, Thanks for the response. Since writing the original email, I've received additional information. WebHDFS does redirect to you the datanode containing the first block are you are requesting. This can be abused to query data locality information but it is inefficient. Since getFileBloc

Re: Data Locality with WebHDFS

2014-03-19 Thread Steve Loughran
1. All the specifics of Hadoop's operations are hidden in the source. That's a get-out clause of OSS, I know, but sometimes it's the clearest. 2. For webhdfs I suspect it picks a local node with the data -you'd have to experiment to make sure 3. If webhdfs is missing fetaures, I'm s

Data Locality with WebHDFS

2014-03-17 Thread RJ Nowling
Hi all, I sent an email to user@ but no one there was able to answer my question. I hope you don't mind me emailing hdfs-dev@ about it. I'm submitting a proposal to Google Summer of Code to add support for HDFS to Disco, an Erlang MapReduce system. We're looking at using WebHDFS. As with Hadoo