Hi, Thanks for your reply. Have been following links for the past two days now. Finally got hadoop natively compiled. Let's see if that solves the problem. Yes, increasing the memory was on my list but i think i tried that, didn't work.
Memory can be issue as it is working perfectly fine for queries with smaller result sets. Also it sometimes works for bigger queries as well. Really strange behavior...Thanks for the tip again. Regards, Sarfraz Rasheed Ramay (DIT) Dublin, Ireland. On Fri, Jul 25, 2014 at 1:51 AM, Juan Martin Pampliega <jpampli...@gmail.com > wrote: > Hi, > The actual useful part of the error is: > > Execution Error, return code 2 from > org.apache.hadoop.hive.ql.exec.mr.MapRedTask > > If you do a search for this plus "EC2" in Google you will find a couple of > results that point to memory exhaustion issues. You should try increasing > the configurated memory size. > > Since you are using a t2.micro you should really try using a bigger Amazon > instance size. This might probably be a lot more useful than trying > different configurations. > On Jul 24, 2014 7:08 AM, "Sarfraz Ramay" <sarfraz.ra...@gmail.com> wrote: > >> Can anyone please help with this ? >> >> [image: Inline image 1] >> >> >> i followed the advice here >> http://stackoverflow.com/questions/20390217/mapreduce-job-in-headless-environment-fails-n-times-due-to-am-container-exceptio >> >> and added to mapred-site.xml following properties but still getting the >> same error. >> >> <property> >> <name>mapred.child.java.opts</name> >> <value>-Djava.awt.headless=true</value></property><!-- add headless to >> default -Xmx1024m --><property> >> <name>yarn.app.mapreduce.am.command-opts</name> >> <value>-Djava.awt.headless=true -Xmx1024m</value></property><property> >> <name>yarn.app.mapreduce.am.admin-command-opts</name> >> <value>-Djava.awt.headless=true</value></property >> >> >> >> Regards, >> Sarfraz Rasheed Ramay (DIT) >> Dublin, Ireland. >> >> >> On Tue, Jul 22, 2014 at 8:19 AM, Sarfraz Ramay <sarfraz.ra...@gmail.com> >> wrote: >> >>> Hi, >>> >>> I am using Hive 0.13.1 and Hadoop 2-2.0 on amazon EC2 t2.micro >>> instances. I have 4 instances, master has the namenode and yarn, >>> secondarynode is a separate instance and two slaves are on separate >>> instances each. >>> >>> It was working fine till now but it started to break when i tried to run >>> the following query on tpch generated 3GB data. same worked ok on 1GB >>> >>> SELECT >>> l_orderkey >>> , sum(l_extendedprice*(1-l_discount)) as revenue >>> , o_orderdate >>> , o_shippriority >>> FROM >>> customer c JOIN orders o >>> ON (c.c_custkey = o.o_custkey) >>> JOIN lineitem l >>> on (l.l_orderkey = o.o_orderkey) >>> WHERE >>> o_orderdate < '1995-03-15' and l_shipdate > '1995-03-15' >>> AND c.c_mktsegment = 'AUTOMOBILE' >>> GROUP BY >>> l_orderkey, o_orderdate, o_shippriority >>> HAVING >>> sum(l_extendedprice*(1-l_discount)) > 38500 --average revenue >>> --LIMIT 10; >>> >>> i have tried many things but nothing seems to work. I am also attaching >>> my mapred-site.xml and yarn-site.xml files for reference plus the error >>> log. I have also tried to limit the memory settings in mapred-site.xml and >>> yarn-site but nothing seems to be working. For full log details please find >>> attached hive.log file. Please help! >>> >>> Hadoop job information for Stage-7: number of mappers: 9; number of >>> reducers: 0 >>> 2014-07-22 06:39:31,643 Stage-7 map = 0%, reduce = 0% >>> 2014-07-22 06:39:43,940 Stage-7 map = 6%, reduce = 0%, Cumulative CPU >>> 5.34 sec >>> 2014-07-22 06:39:45,002 Stage-7 map = 11%, reduce = 0%, Cumulative CPU >>> 6.94 sec >>> 2014-07-22 06:40:08,373 Stage-7 map = 17%, reduce = 0%, Cumulative CPU >>> 12.6 sec >>> 2014-07-22 06:40:10,417 Stage-7 map = 22%, reduce = 0%, Cumulative CPU >>> 14.06 sec >>> 2014-07-22 06:40:22,732 Stage-7 map = 28%, reduce = 0%, Cumulative CPU >>> 24.46 sec >>> 2014-07-22 06:40:25,843 Stage-7 map = 33%, reduce = 0%, Cumulative CPU >>> 25.74 sec >>> 2014-07-22 06:40:33,039 Stage-7 map = 44%, reduce = 0%, Cumulative CPU >>> 33.32 sec >>> 2014-07-22 06:40:38,709 Stage-7 map = 56%, reduce = 0%, Cumulative CPU >>> 37.19 sec >>> 2014-07-22 06:41:07,648 Stage-7 map = 61%, reduce = 0%, Cumulative CPU >>> 42.83 sec >>> 2014-07-22 06:41:15,900 Stage-7 map = 56%, reduce = 0%, Cumulative CPU >>> 39.49 sec >>> 2014-07-22 06:41:27,299 Stage-7 map = 67%, reduce = 0%, Cumulative CPU >>> 46.07 sec >>> 2014-07-22 06:41:28,342 Stage-7 map = 56%, reduce = 0%, Cumulative CPU >>> 40.9 sec >>> 2014-07-22 06:41:43,753 Stage-7 map = 61%, reduce = 0%, Cumulative CPU >>> 42.84 sec >>> 2014-07-22 06:41:45,801 Stage-7 map = 100%, reduce = 0%, Cumulative CPU >>> 37.19 sec >>> MapReduce Total cumulative CPU time: 37 seconds 190 msec >>> Ended Job = job_1406011031680_0002 with errors >>> Error during job, obtaining debugging information... >>> Job Tracking URL: >>> http://ec2-54-77-76-145.eu-west-1.compute.amazonaws.com:8088/proxy/application_1406011031680_0002/ >>> Examining task ID: task_1406011031680_0002_m_000001 (and more) from job >>> job_1406011031680_0002 >>> Examining task ID: task_1406011031680_0002_m_000005 (and more) from job >>> job_1406011031680_0002 >>> >>> Task with the most failures(4): >>> ----- >>> Task ID: >>> task_1406011031680_0002_m_000008 >>> >>> URL: >>> >>> http://ec2-54-77-76-145.eu-west-1.compute.amazonaws.com:8088/taskdetails.jsp?jobid=job_1406011031680_0002&tipid=task_1406011031680_0002_m_000008 >>> ----- >>> Diagnostic Messages for this Task: >>> Exception from container-launch: >>> org.apache.hadoop.util.Shell$ExitCodeException: >>> at org.apache.hadoop.util.Shell.runCommand(Shell.java:464) >>> at org.apache.hadoop.util.Shell.run(Shell.java:379) >>> at >>> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:589) >>> at >>> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195) >>> at >>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:283) >>> at >>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:79) >>> at java.util.concurrent.FutureTask.run(FutureTask.java:262) >>> at >>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >>> at >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >>> at java.lang.Thread.run(Thread.java:744) >>> >>> >>> FAILED: Execution Error, return code 2 from >>> org.apache.hadoop.hive.ql.exec.mr.MapRedTask >>> MapReduce Jobs Launched: >>> Job 0: Map: 3 Reduce: 1 Cumulative CPU: 24.58 sec HDFS Read: >>> 593821601 HDFS Write: 14518009 SUCCESS >>> Job 1: Map: 9 Cumulative CPU: 37.19 sec HDFS Read: 1342219615 HDFS >>> Write: 821879 FAIL >>> Total MapReduce CPU Time Spent: 1 minutes 1 seconds 770 msec >>> hive (default)> exit; >>> >>> >>> >>> Regards, >>> Sarfraz Rasheed Ramay (DIT) >>> Dublin, Ireland. >>> >> >>
