mapred.local.dir can definitely work with multiple paths, but it's a bit strict about the format. You should have one or more paths, separated by commas and no whitespace. e.g. "/disk1/mapred,/disk2/mapred". If you've got them on new lines, etc, then it might try to interpret that as part of one of the paths. Could this be your issue? If you paste your hadoop-site.xml in to an email we could take a look.
- Aaron On Wed, Jun 10, 2009 at 2:41 AM, Harish Mallipeddi < [email protected]> wrote: > Hi, > > After some amount of debugging, I narrowed down the problem. I'd set > multiple paths (multiple disks mounted at different endpoints) for the * > mapred.local.dir* setting. When I changed this to just one path, everything > started working - previously the job jar and the job xml weren't being > created under the local/taskTracker/jobCache folder. > > Any idea why? Am I doing something wrong? I've read somewhere else that > specifying multiple disks for mapred.local.dir is important to increase > disk > bandwidth so that the map outputs get written faster to the local disks on > the tasktracker nodes. > > Cheers, > Harish > > On Tue, Jun 9, 2009 at 5:21 PM, Harish Mallipeddi < > [email protected]> wrote: > > > I setup a new cluster (1 namenode + 2 datanodes). I'm trying to run the > > GenericMRLoadGenerator program from hadoop-0.20.0-test.jar. But I keep > > getting the ClassNotFoundException. Any reason why this would be > happening? > > It seems to me like the tasktrackers cannot find the class files from the > > hadoop program but when you do "hadoop jar", it will automatically ship > the > > job jar file to all the tasktracker nodes and put them in the classpath? > > > > $ *bin/hadoop jar hadoop-0.20.0-test.jar > loadgen*-Dtest.randomtextwrite.bytes_per_map=1048576 > > -Dhadoop.sort.reduce.keep.percent=50.0 -outKey org.apache.hadoop.io.Text > > -outValue org.apache.hadoop.io.Text > > > > No input path; ignoring InputFormat > > Job started: Tue Jun 09 03:16:30 PDT 2009 > > 09/06/09 03:16:30 INFO mapred.JobClient: Running job: > job_200906090316_0001 > > 09/06/09 03:16:31 INFO mapred.JobClient: map 0% reduce 0% > > 09/06/09 03:16:40 INFO mapred.JobClient: Task Id : > > attempt_200906090316_0001_m_000000_0, Status : FAILED > > java.io.IOException: Split class > > > org.apache.hadoop.mapred.GenericMRLoadGenerator$IndirectInputFormat$IndirectSplit > > not found > > at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:324) > > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305) > > at org.apache.hadoop.mapred.Child.main(Child.java:170) > > *Caused by: java.lang.ClassNotFoundException: > > > org.apache.hadoop.mapred.GenericMRLoadGenerator$IndirectInputFormat$IndirectSplit > > * > > at java.net.URLClassLoader$1.run(URLClassLoader.java:200) > > at java.security.AccessController.doPrivileged(Native Method) > > at java.net.URLClassLoader.findClass(URLClassLoader.java:188) > > at java.lang.ClassLoader.loadClass(ClassLoader.java:307) > > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) > > at java.lang.ClassLoader.loadClass(ClassLoader.java:252) > > at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320) > > at java.lang.Class.forName0(Native Method) > > at java.lang.Class.forName(Class.java:247) > > at > > > org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:761) > > at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:321) > > ... 2 more > > [more of the same exception from different map tasks...I've removed them] > > > > Actually the same thing happens even if I run the PiEstimator program > from > > hadoop-0.20.0-examples.jar. > > > > Thanks, > > > > -- > > Harish Mallipeddi > > http://blog.poundbang.in > > > > > > -- > Harish Mallipeddi > http://blog.poundbang.in >
