On Wednesday 26 May 2010 04:38 PM, Saurabh Agarwal wrote:
Hii.. I am toying around with hadoop configuration. I am trying to replace HDFS with a common nfsmount, I only have map tasks. so intermediate outputs need not be communicated!!! So I want is there a way to make the temp directory local to the nodes and place job conf object and jar in a nfs mount so all the nodes can access it.. Saurabh Agarwal
Ideally you can do it, because MapReduce uses FileSystem APIs everywhere, but you may find some quirks.
OTOH, it is a very very bad idea and highly discouraged to run MapReduce on NFS - as soon as the number of nodes and thus tasks scales up, NFS will become bottlenecked and tasks/jobs will start failing with hard to debug failures.
+Vinod
