Hi All,
We have not been able to get what is in the subject line to run. This is
on hive 0.14. While pulling a billion row table from Oracle using 12
splits on the primary key, each job continually runs out of memory such as
below...
15/03/13 00:22:23 INFO mapreduce.Job: Task Id :
attempt_1426097251374_0011_m_000011_0, Status : FAILED
Container [pid=27919,containerID=container_1426097251374_0011_01_000013] is
running beyond physical memory limits. Current usage: 513.5 MB of 512 MB
physical memory used; 879.3 MB of 1.0 GB virtual memory used. Killing
container.
Dump of the process-tree for container_1426097251374_0011_01_000013 :
|- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS)
SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
|- 28078 27919 27919 27919 (java) 63513 834 912551936 131129
/usr/jdk64/jdk1.7.0_45/bin/java -server -XX:NewRatio=8
-Djava.net.preferIPv4Stack=true -Dhdp.version=2.2.0.0-2041 -Xmx410m
-Djava.io.tmpdir=/mnt/hdfs/hadoop/yarn/local/usercache/hdfs/appcache/application_1426097251374_0011/container_1426097251374_0011_01_000013/tmp
-Dlog4j.configuration=container-log4j.properties
-Dyarn.app.container.log.dir=/mnt/hdfs/hadoop/yarn/log/application_1426097251374_0011/container_1426097251374_0011_01_000013
-Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA
org.apache.hadoop.mapred.YarnChild 172.27.2.57 52335
attempt_1426097251374_0011_m_000011_0 13
|- 27919 27917 27919 27919 (bash) 1 2 9424896 317 /bin/bash -c
/usr/jdk64/jdk1.7.0_45/bin/java -server -XX:NewRatio=8
-Djava.net.preferIPv4Stack=true -Dhdp.version=2.2.0.0-2041 -Xmx410m
-Djava.io.tmpdir=/mnt/hdfs/hadoop/yarn/local/usercache/hdfs/appcache/application_1426097251374_0011/container_1426097251374_0011_01_000013/tmp
-Dlog4j.configuration=container-log4j.properties
-Dyarn.app.container.log.dir=/mnt/hdfs/hadoop/yarn/log/application_1426097251374_0011/container_1426097251374_0011_01_000013
-Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA
org.apache.hadoop.mapred.YarnChild 172.27.2.57 52335
attempt_1426097251374_0011_m_000011_0 13
1>/mnt/hdfs/hadoop/yarn/log/application_1426097251374_0011/container_1426097251374_0011_01_000013/stdout
2>/mnt/hdfs/hadoop/yarn/log/application_1426097251374_0011/container_1426097251374_0011_01_000013/stderr
Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
We have tried several different sizes for various switches, but the job
always fails.
Is this simply a function of the data, or is there another issue?
Thanks,
Steve
