Hej
I am running an aplication based on Nutch in a cluster with 1 master and 6
slaves in Amazon (with the same instances for all of them with 1.7GB RAM
memory)
My configuration is the following:
HADOOP_HEAPSIZE=1300
HADOOP_NAMENODE_OPTS=-Xmx400m
HADOOP_SECONDARYNAMENODE_OPTS=-Xmx400m
HADOOP_JOBTRACKER_OPTS=-Xmx400m
dfs.replication=3
mapred.map.tasks=6
mapred.reduce.tasks=6
mapred.child.java.opts=-Xmx950m
But in the second depth fetch, I got the following errors in some instances
(while the other ones seems they fetched correctly) :
2010-02-10 03:18:31,185 FATAL fetcher.Fetcher - java.io.IOException: Spill
failed
2010-02-10 03:18:31,185 FATAL fetcher.Fetcher - at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:822)
2010-02-10 03:18:31,185 FATAL fetcher.Fetcher - at
org.apache.hadoop.mapred.MapTask$OldOutputCollector.collect(MapTask.java:466)
2010-02-10 03:18:31,185 FATAL fetcher.Fetcher - at
org.apache.nutch.fetcher.Fetcher$FetcherThread.output(Fetcher.java:907)
2010-02-10 03:18:31,185 FATAL fetcher.Fetcher - at
org.apache.nutch.fetcher.Fetcher$FetcherThread.run(Fetcher.java:670)
2010-02-10 03:18:31,185 FATAL fetcher.Fetcher - Caused by:
java.io.IOException: Cannot run program "bash": java.io.IOException:
error=12, Cannot allocate memory
2010-02-10 03:18:31,185 FATAL fetcher.Fetcher - at
java.lang.ProcessBuilder.start(ProcessBuilder.java:459)
2010-02-10 03:18:31,185 FATAL fetcher.Fetcher - at
org.apache.hadoop.util.Shell.runCommand(Shell.java:149)
2010-02-10 03:18:31,185 FATAL fetcher.Fetcher - at
org.apache.hadoop.util.Shell.run(Shell.java:134)
2010-02-10 03:18:31,185 FATAL fetcher.Fetcher - at
org.apache.hadoop.fs.DF.getAvailable(DF.java:73)
2010-02-10 03:18:31,185 FATAL fetcher.Fetcher - at
org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:329)
2010-02-10 03:18:31,185 FATAL fetcher.Fetcher - at
org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:124)
2010-02-10 03:18:31,185 FATAL fetcher.Fetcher - at
org.apache.hadoop.mapred.MapOutputFile.getSpillFileForWrite(MapOutputFile.java:107)
2010-02-10 03:18:31,185 FATAL fetcher.Fetcher - at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1183)
2010-02-10 03:18:31,185 FATAL fetcher.Fetcher - at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.access$1800(MapTask.java:648)
2010-02-10 03:18:31,185 FATAL fetcher.Fetcher - at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer$SpillThread.run(MapTask.java:1135)
2010-02-10 03:18:31,185 FATAL fetcher.Fetcher - Caused by:
java.io.IOException: java.io.IOException: error=12, Cannot allocate memory
2010-02-10 03:18:31,185 FATAL fetcher.Fetcher - at
java.lang.UNIXProcess.<init>(UNIXProcess.java:148)
2010-02-10 03:18:31,185 FATAL fetcher.Fetcher - at
java.lang.ProcessImpl.start(ProcessImpl.java:65)
2010-02-10 03:18:31,185 FATAL fetcher.Fetcher - at
java.lang.ProcessBuilder.start(ProcessBuilder.java:452)
2010-02-10 03:18:31,185 FATAL fetcher.Fetcher - ... 9 more
.
.
.
.
.
2010-02-10 03:18:31,463 WARN mapred.TaskTracker - Error running child
java.io.IOException: Spill failed
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1085)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:359)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
at org.apache.hadoop.mapred.Child.main(Child.java:170)
Caused by: java.io.IOException: Cannot run program "bash":
java.io.IOException: error=12, Cannot allocate memory
at java.lang.ProcessBuilder.start(ProcessBuilder.java:459)
at org.apache.hadoop.util.Shell.runCommand(Shell.java:149)
at org.apache.hadoop.util.Shell.run(Shell.java:134)
at org.apache.hadoop.fs.DF.getAvailable(DF.java:73)
at
org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:329)
at
org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:124)
at
org.apache.hadoop.mapred.MapOutputFile.getSpillFileForWrite(MapOutputFile.java:107)
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1183)
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.access$1800(MapTask.java:648)
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer$SpillThread.run(MapTask.java:1135)
Caused by: java.io.IOException: java.io.IOException: error=12, Cannot
allocate memory
at java.lang.UNIXProcess.<init>(UNIXProcess.java:148)
at java.lang.ProcessImpl.start(ProcessImpl.java:65)
at java.lang.ProcessBuilder.start(ProcessBuilder.java:452)
... 9 more
Any idea??
Thanks in advance :)
--
View this message in context:
http://old.nabble.com/Spill-failed-tp27527092p27527092.html
Sent from the Hadoop core-dev mailing list archive at Nabble.com.
