I had a similar problem until I set this parameter to 1 (although 3 seems to work fine too).
There is an explanation somewhere on the web. Basically, if you run 20 tasks and the garbage collector cannot catch up with accumulated garbage, the java process grows too big so when it finally decides to fork a new process, the forked process and the original one take up too much memory. Or something like that. I imagine the same kind thing would still happen even when setting reuse to 1 if a task happens to consume a lot of memory and CPU cycles, so something else must be happening here. Perhaps, there is a memory 'leak' and some 'garbage' never gets collected between task executions. On Thu, Apr 7, 2011 at 6:53 PM, Steven Wong <sw...@netflix.com> wrote: > When the following query was run with mapred.job.reuse.jvm.num.tasks=20, > some of the map tasks failed with “Error: Java heap space”, causing the job > to fail. After changing to mapred.job.reuse.jvm.num.tasks=1, the job > succeeded. > > > > FROM ( > > FROM intable1 > > SELECT acct_id, esn) b > > JOIN ( > > FROM intable2 > > SELECT acct_id, xid, devtype_id, esn, other_properties['cdndldist'] AS > cdndldist > > WHERE (dateint>=20110201 AND dateint<=20110228) > > AND (other_properties['cdndldist'] is not null) > > AND (client_msg_type='endplay') > > AND (devtype_id=272 OR devtype_id=129 OR devtype_id=12 OR devtype_id=13 > OR devtype_id=14)) c > > ON (b.acct_id=c.acct_id) AND (b.esn=c.esn) > > INSERT OVERWRITE TABLE outtable PARTITION (dateint='20110201-20110228') > > SELECT b.acct_id, c.xid, c.devtype_id, c.esn, c.cdndldist; > > > > I’m on a pre-release version of Hive 0.7, with Hadoop 0.20. Does anyone > know about any Hive/Hadoop issue that may be related to this? > > > > Thanks. > > Steven >
