Can anyone please help with this ? [image: Inline image 1]
i followed the advice here http://stackoverflow.com/questions/20390217/mapreduce-job-in-headless-environment-fails-n-times-due-to-am-container-exceptio and added to mapred-site.xml following properties but still getting the same error. <property> <name>mapred.child.java.opts</name> <value>-Djava.awt.headless=true</value></property><!-- add headless to default -Xmx1024m --><property> <name>yarn.app.mapreduce.am.command-opts</name> <value>-Djava.awt.headless=true -Xmx1024m</value></property><property> <name>yarn.app.mapreduce.am.admin-command-opts</name> <value>-Djava.awt.headless=true</value></property Regards, Sarfraz Rasheed Ramay (DIT) Dublin, Ireland. On Tue, Jul 22, 2014 at 8:19 AM, Sarfraz Ramay <sarfraz.ra...@gmail.com> wrote: > Hi, > > I am using Hive 0.13.1 and Hadoop 2-2.0 on amazon EC2 t2.micro instances. > I have 4 instances, master has the namenode and yarn, secondarynode is a > separate instance and two slaves are on separate instances each. > > It was working fine till now but it started to break when i tried to run > the following query on tpch generated 3GB data. same worked ok on 1GB > > SELECT > l_orderkey > , sum(l_extendedprice*(1-l_discount)) as revenue > , o_orderdate > , o_shippriority > FROM > customer c JOIN orders o > ON (c.c_custkey = o.o_custkey) > JOIN lineitem l > on (l.l_orderkey = o.o_orderkey) > WHERE > o_orderdate < '1995-03-15' and l_shipdate > '1995-03-15' > AND c.c_mktsegment = 'AUTOMOBILE' > GROUP BY > l_orderkey, o_orderdate, o_shippriority > HAVING > sum(l_extendedprice*(1-l_discount)) > 38500 --average revenue > --LIMIT 10; > > i have tried many things but nothing seems to work. I am also attaching my > mapred-site.xml and yarn-site.xml files for reference plus the error log. I > have also tried to limit the memory settings in mapred-site.xml and > yarn-site but nothing seems to be working. For full log details please find > attached hive.log file. Please help! > > Hadoop job information for Stage-7: number of mappers: 9; number of > reducers: 0 > 2014-07-22 06:39:31,643 Stage-7 map = 0%, reduce = 0% > 2014-07-22 06:39:43,940 Stage-7 map = 6%, reduce = 0%, Cumulative CPU > 5.34 sec > 2014-07-22 06:39:45,002 Stage-7 map = 11%, reduce = 0%, Cumulative CPU > 6.94 sec > 2014-07-22 06:40:08,373 Stage-7 map = 17%, reduce = 0%, Cumulative CPU > 12.6 sec > 2014-07-22 06:40:10,417 Stage-7 map = 22%, reduce = 0%, Cumulative CPU > 14.06 sec > 2014-07-22 06:40:22,732 Stage-7 map = 28%, reduce = 0%, Cumulative CPU > 24.46 sec > 2014-07-22 06:40:25,843 Stage-7 map = 33%, reduce = 0%, Cumulative CPU > 25.74 sec > 2014-07-22 06:40:33,039 Stage-7 map = 44%, reduce = 0%, Cumulative CPU > 33.32 sec > 2014-07-22 06:40:38,709 Stage-7 map = 56%, reduce = 0%, Cumulative CPU > 37.19 sec > 2014-07-22 06:41:07,648 Stage-7 map = 61%, reduce = 0%, Cumulative CPU > 42.83 sec > 2014-07-22 06:41:15,900 Stage-7 map = 56%, reduce = 0%, Cumulative CPU > 39.49 sec > 2014-07-22 06:41:27,299 Stage-7 map = 67%, reduce = 0%, Cumulative CPU > 46.07 sec > 2014-07-22 06:41:28,342 Stage-7 map = 56%, reduce = 0%, Cumulative CPU > 40.9 sec > 2014-07-22 06:41:43,753 Stage-7 map = 61%, reduce = 0%, Cumulative CPU > 42.84 sec > 2014-07-22 06:41:45,801 Stage-7 map = 100%, reduce = 0%, Cumulative CPU > 37.19 sec > MapReduce Total cumulative CPU time: 37 seconds 190 msec > Ended Job = job_1406011031680_0002 with errors > Error during job, obtaining debugging information... > Job Tracking URL: > http://ec2-54-77-76-145.eu-west-1.compute.amazonaws.com:8088/proxy/application_1406011031680_0002/ > Examining task ID: task_1406011031680_0002_m_000001 (and more) from job > job_1406011031680_0002 > Examining task ID: task_1406011031680_0002_m_000005 (and more) from job > job_1406011031680_0002 > > Task with the most failures(4): > ----- > Task ID: > task_1406011031680_0002_m_000008 > > URL: > > http://ec2-54-77-76-145.eu-west-1.compute.amazonaws.com:8088/taskdetails.jsp?jobid=job_1406011031680_0002&tipid=task_1406011031680_0002_m_000008 > ----- > Diagnostic Messages for this Task: > Exception from container-launch: > org.apache.hadoop.util.Shell$ExitCodeException: > at org.apache.hadoop.util.Shell.runCommand(Shell.java:464) > at org.apache.hadoop.util.Shell.run(Shell.java:379) > at > org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:589) > at > org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:283) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:79) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) > > > FAILED: Execution Error, return code 2 from > org.apache.hadoop.hive.ql.exec.mr.MapRedTask > MapReduce Jobs Launched: > Job 0: Map: 3 Reduce: 1 Cumulative CPU: 24.58 sec HDFS Read: > 593821601 HDFS Write: 14518009 SUCCESS > Job 1: Map: 9 Cumulative CPU: 37.19 sec HDFS Read: 1342219615 HDFS > Write: 821879 FAIL > Total MapReduce CPU Time Spent: 1 minutes 1 seconds 770 msec > hive (default)> exit; > > > > Regards, > Sarfraz Rasheed Ramay (DIT) > Dublin, Ireland. >
