Yeah, it works!
Bhupesh and Jason, thanks a lot!
Jasmine
----- Original Message -----
From: "jason hadoop" <[email protected]>
To: <[email protected]>
Sent: Wednesday, April 29, 2009 11:19 PM
Subject: Re: What's the local heap size of Hadoop? How to increase it?
you can also specify it on the command line of your hadoop job
hadoop jar jarfile [main class] -D mapred.child.java.opts=-Xmx800M other
arguments
Note: there is a space between the -D and mapred, and the -D has to come
after the main class specification.
This parameter may also be specified via
conf.set("mapred.child.java.opts","-Xmx800m"); before submitting the job.
On Wed, Apr 29, 2009 at 2:59 PM, Bhupesh Bansal
<[email protected]>wrote:
Hey ,
Try adding
<property>
<name>mapred.child.java.opts</name>
<value>-Xmx800M -server</value>
</property>
With the right JVM size in your hadoop-site.xml , you will have to copy
this
to all mapred nodes and restart the cluster.
Best
Bhupesh
On 4/29/09 2:03 PM, "Jasmine (Xuanjing) Huang" <[email protected]>
wrote:
> Hi, there,
>
> What's the local heap size of Hadoop? I have tried to load a local
> cache
> file which is composed of 500,000 short phrase, but the task failed.
> The
> output of Hadoop looks like(com.aliasi.dict.ExactDictionaryChunker is a
> third-party jar package, and the records is organized as a trie
struction):
>
> java.lang.OutOfMemoryError: Java heap space
> at java.util.HashMap.addEntry(HashMap.java:753)
> at java.util.HashMap.put(HashMap.java:385)
> at
> com.aliasi.dict.ExactDictionaryChunker$TrieNode.getOrCreateDaughter(Ex
> actDictionaryChunker.java:476)
> at
> com.aliasi.dict.ExactDictionaryChunker$TrieNode.add(ExactDictionaryChu
> nker.java:484)
>
> When I reduce the total record number to 30,000. My mapreduce job
> became
> succeed. So, I have a question, What's the local heap size of Hadoop's
Java
> Virtual Machine? How to increase it?
>
> Best,
> Jasmine
>
--
Alpha Chapters of my book on Hadoop are available
http://www.apress.com/book/view/9781430219422
