you can also specify it on the command line of your hadoop job
hadoop jar jarfile [main class] -D mapred.child.java.opts=-Xmx800M other
arguments
Note: there is a space between the -D and mapred, and the -D has to come
after the main class specification.
This parameter may also be specified via
conf.set("mapred.child.java.opts","-Xmx800m"); before submitting the job.
On Wed, Apr 29, 2009 at 2:59 PM, Bhupesh Bansal <[email protected]>wrote:
> Hey ,
>
> Try adding
>
> <property>
> <name>mapred.child.java.opts</name>
> <value>-Xmx800M -server</value>
> </property>
>
> With the right JVM size in your hadoop-site.xml , you will have to copy
> this
> to all mapred nodes and restart the cluster.
>
> Best
> Bhupesh
>
>
>
> On 4/29/09 2:03 PM, "Jasmine (Xuanjing) Huang" <[email protected]>
> wrote:
>
> > Hi, there,
> >
> > What's the local heap size of Hadoop? I have tried to load a local cache
> > file which is composed of 500,000 short phrase, but the task failed. The
> > output of Hadoop looks like(com.aliasi.dict.ExactDictionaryChunker is a
> > third-party jar package, and the records is organized as a trie
> struction):
> >
> > java.lang.OutOfMemoryError: Java heap space
> > at java.util.HashMap.addEntry(HashMap.java:753)
> > at java.util.HashMap.put(HashMap.java:385)
> > at
> > com.aliasi.dict.ExactDictionaryChunker$TrieNode.getOrCreateDaughter(Ex
> > actDictionaryChunker.java:476)
> > at
> > com.aliasi.dict.ExactDictionaryChunker$TrieNode.add(ExactDictionaryChu
> > nker.java:484)
> >
> > When I reduce the total record number to 30,000. My mapreduce job became
> > succeed. So, I have a question, What's the local heap size of Hadoop's
> Java
> > Virtual Machine? How to increase it?
> >
> > Best,
> > Jasmine
> >
>
>
--
Alpha Chapters of my book on Hadoop are available
http://www.apress.com/book/view/9781430219422
