Hi Todd I just made a simple execution with 'yarn-client’ and it seems to work with my hive and spark. I have been using the pre-built version of Spark 1.4.1, so in my case I did not have to build it myself.
<http://www.taiger.com/> SOFIA PANAGIOTIDI www.taiger.com <http://www.taiger.com/> - +34 914497284 sofia.panagiot...@taiger.com - Taiger's everywhere. I'm in Madrid > On 12 Jan 2016, at 03:54, Todd <bit1...@163.com> wrote: > > Hi Sofia, > Thank you for the detailed guide. > I confirmed that there are compatibility issues between Hive 1.2.1 and Spark > 1.5.X. Hive 1.2.1 is by default built against spark 1.3.1 and compatible with > spark 1.4.1,but doesn't work with 1.5.X. > > Basically,I am using somewhat the same configuration as yours,but I am using > yarn-client as the spark.master. > Did you try to use yarn-client as the spark.master? > > I am building spark1.3.1 without hive and hive-thriftserver. I will get back > to you when I do the test, > > Thank you, Sofia. > > > > > At 2016-01-11 23:56:32, "Sofia" <sofia.panagiot...@taiger.com> wrote: > From what I have seen you need to use a Spark version that does not contain > the Hive jars. (maybe build one as described at the page). > Maybe someone can respond to the compatibility issue better than me. > > However, this is how I managed to make hive work with Spark > > I had similar issues with the downloaded Spark 1.5.2 and in the end I used > the prebuilt Spark 1.4.1 with my Hive 1.2.1 (and Hadoop 2.7.1). > When doing the setup I used the configurations below: > > hive-env.sh > > export SPARK_HOME=/usr/local/spark > export HADOOP_HOME=/usr/local/hadoop > > hive-site.xml > > <property> > <name>hive.execution.engine</name> > <value>spark</value> > </property> > <property> > <name>spark.master</name> > <value>spark://spark-master:7077</value> > <spark://spark-master:7077</value>> > </property> > > > spark-env.sh > > export HADOOP_HOME=/usr/local/hadoop > export PATH=$PATH:$HADOOP_HOME/bin > export SPARK_HOME=/usr/local/spark > > export SPARK_DIST_CLASSPATH=`hadoop classpath` > export SPARK_CLASSPATH=$CLASSPATH > > > I don’t know the answer to your issue exactly as at some point I faced the > same, but I hope this might help. > Cheers > > >> On 11 Jan 2016, at 11:39, Todd <bit1...@163.com <mailto:bit1...@163.com>> >> wrote: >> >> >> Thank you, Sofia. >> >> From the log, looks it is java.lang.AbstractMethodError that leads to the >> job failure. >> I am using Hive 1.2.1+Spark 1.5.2, is this a compatibility issue? >> >> >> >> >> 16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11 >> 17:42:06 INFO log.PerfLogger: <PERFLOG method=SparkBuildPlan >> from=org.apache.hadoop.hive.ql.exec.spark.SparkPlanGenerator> >> 16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11 >> 17:42:06 INFO log.PerfLogger: <PERFLOG method=SparkCreateTran.Map 1 >> from=org.apache.hadoop.hive.ql.exec.spark.SparkPlanGenerator> >> 16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11 >> 17:42:06 INFO Configuration.deprecation: mapred.task.is.map is deprecated. >> Instead, use mapreduce.task.ismap >> 16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11 >> 17:42:06 INFO exec.Utilities: Processing alias t1 >> 16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11 >> 17:42:06 INFO exec.Utilities: Adding input file >> hdfs://hadoop.bit.com:9000/user/hive/warehouse/db_1.db/t1 >> <hdfs://hadoop.bit.com:9000/user/hive/warehouse/db_1.db/t1> >> 16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11 >> 17:42:06 INFO log.PerfLogger: <PERFLOG method=serializePlan >> from=org.apache.hadoop.hive.ql.exec.Utilities> >> 16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11 >> 17:42:06 INFO exec.Utilities: Serializing MapWork via kryo >> 16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11 >> 17:42:06 INFO log.PerfLogger: </PERFLOG method=serializePlan >> start=1452505326307 end=1452505326332 duration=25 >> from=org.apache.hadoop.hive.ql.exec.Utilities> >> 16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11 >> 17:42:06 INFO storage.MemoryStore: ensureFreeSpace(505800) called with >> curMem=0, maxMem=278302556 >> 16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11 >> 17:42:06 INFO storage.MemoryStore: Block broadcast_0 stored as values in >> memory (estimated size 493.9 KB, free 264.9 MB) >> 16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11 >> 17:42:06 INFO storage.MemoryStore: ensureFreeSpace(42143) called with >> curMem=505800, maxMem=278302556 >> 16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11 >> 17:42:06 INFO storage.MemoryStore: Block broadcast_0_piece0 stored as bytes >> in memory (estimated size 41.2 KB, free 264.9 MB) >> 16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11 >> 17:42:06 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory >> on 10.12.167.42:38883 (size: 41.2 KB, free: 265.4 MB) >> 16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11 >> 17:42:06 ERROR util.Utils: uncaught error in thread SparkListenerBus, >> stopping SparkContext >> 16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: >> java.lang.AbstractMethodError >> 16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: at >> org.apache.spark.scheduler.SparkListenerBus$class.onPostEvent(SparkListenerBus.scala:62) >> 16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: at >> org.apache.spark.scheduler.LiveListenerBus.onPostEvent(LiveListenerBus.scala:31) >> 16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: at >> org.apache.spark.scheduler.LiveListenerBus.onPostEvent(LiveListenerBus.scala:31) >> 16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: at >> org.apache.spark.util.ListenerBus$class.postToAll(ListenerBus.scala:56) >> 16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: at >> org.apache.spark.util.AsynchronousListenerBus.postToAll(AsynchronousListenerBus.scala:37) >> 16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: at >> org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1.apply$mcV$sp(AsynchronousListenerBus.scala:79) >> 16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: at >> org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1136) >> 16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: at >> org.apache.spark.util.AsynchronousListenerBus$$anon$1.run(AsynchronousListenerBus.scala:63) >> 16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11 >> 17:42:06 INFO spark.SparkContext: Created broadcast 0 from hadoopRDD at >> SparkPlanGenerator.java:188 >> 16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11 >> 17:42:06 INFO handler.ContextHandler: stopped >> o.s.j.s.ServletContextHandler{/metrics/json,null} >> 16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11 >> 17:42:06 INFO handler.ContextHandler: stopped >> o.s.j.s.ServletContextHandler{/stages/stage/kill,null} >> 16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11 >> 17:42:06 INFO handler.ContextHandler: stopped >> o.s.j.s.ServletContextHandler{/api,null} >> 16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11 >> 17:42:06 INFO handler.ContextHandler: stopped >> o.s.j.s.ServletContextHandler{/,null} >> 16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11 >> 17:42:06 INFO handler.ContextHandler: stopped >> o.s.j.s.ServletContextHandler{/static,null} >> 16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11 >> 17:42:06 INFO handler.ContextHandler: stopped >> o.s.j.s.ServletContextHandler{/executors/threadDump/json,null} >> 16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11 >> 17:42:06 INFO handler.ContextHandler: stopped >> o.s.j.s.ServletContextHandler{/executors/threadDump,null} >> 16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11 >> 17:42:06 INFO handler.ContextHandler: stopped >> o.s.j.s.ServletContextHandler{/executors/json,null} >> 16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11 >> 17:42:06 INFO handler.ContextHandler: stopped >> o.s.j.s.ServletContextHandler{/executors,null} >> 16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11 >> 17:42:06 INFO handler.ContextHandler: stopped >> o.s.j.s.ServletContextHandler{/environment/json,null} >> 16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11 >> 17:42:06 INFO handler.ContextHandler: stopped >> o.s.j.s.ServletContextHandler{/environment,null} >> 16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11 >> 17:42:06 INFO handler.ContextHandler: stopped >> o.s.j.s.ServletContextHandler{/storage/rdd/json,null} >> 16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11 >> 17:42:06 INFO handler.ContextHandler: stopped >> o.s.j.s.ServletContextHandler{/storage/rdd,null} >> 16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11 >> 17:42:06 INFO handler.ContextHandler: stopped >> o.s.j.s.ServletContextHandler{/storage/json,null} >> 16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11 >> 17:42:06 INFO handler.ContextHandler: stopped >> o.s.j.s.ServletContextHandler{/storage,null} >> 16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11 >> 17:42:06 INFO handler.ContextHandler: stopped >> o.s.j.s.ServletContextHandler{/stages/pool/json,null} >> 16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11 >> 17:42:06 INFO handler.ContextHandler: stopped >> o.s.j.s.ServletContextHandler{/stages/pool,null} >> 16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11 >> 17:42:06 INFO handler.ContextHandler: stopped >> o.s.j.s.ServletContextHandler{/stages/stage/json,null} >> 16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11 >> 17:42:06 INFO handler.ContextHandler: stopped >> o.s.j.s.ServletContextHandler{/stages/stage,null} >> 16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11 >> 17:42:06 INFO handler.ContextHandler: stopped >> o.s.j.s.ServletContextHandler{/stages/json,null} >> 16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11 >> 17:42:06 INFO handler.ContextHandler: stopped >> o.s.j.s.ServletContextHandler{/stages,null} >> 16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11 >> 17:42:06 INFO handler.ContextHandler: stopped >> o.s.j.s.ServletContextHandler{/jobs/job/json,null} >> 16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11 >> 17:42:06 INFO handler.ContextHandler: stopped >> o.s.j.s.ServletContextHandler{/jobs/job,null} >> 16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11 >> 17:42:06 INFO handler.ContextHandler: stopped >> o.s.j.s.ServletContextHandler{/jobs/json,null} >> 16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11 >> 17:42:06 INFO handler.ContextHandler: stopped >> o.s.j.s.ServletContextHandler{/jobs,null} >> 16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11 >> 17:42:06 INFO log.PerfLogger: </PERFLOG method=SparkCreateTran.Map 1 >> start=1452505326274 end=1452505326824 duration=550 >> from=org.apache.hadoop.hive.ql.exec.spark.SparkPlanGenerator> >> 16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11 >> 17:42:06 INFO log.PerfLogger: <PERFLOG method=SparkCreateTran.Reducer 2 >> from=org.apache.hadoop.hive.ql.exec.spark.SparkPlanGenerator> >> 16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11 >> 17:42:06 INFO log.PerfLogger: <PERFLOG method=serializePlan >> from=org.apache.hadoop.hive.ql.exec.Utilities> >> 16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11 >> 17:42:06 INFO exec.Utilities: Serializing ReduceWork via kryo >> 16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11 >> 17:42:06 INFO ui.SparkUI: Stopped Spark web UI at http://10.12.167.42:4040 >> <http://10.12.167.42:4040/> >> 16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11 >> 17:42:06 INFO scheduler.DAGScheduler: Stopping DAGScheduler >> 16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11 >> 17:42:06 INFO cluster.YarnClientSchedulerBackend: Shutting down all executors >> 16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11 >> 17:42:06 INFO cluster.YarnClientSchedulerBackend: Interrupting monitor thread >> 16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11 >> 17:42:06 INFO cluster.YarnClientSchedulerBackend: Asking each executor to >> shut down >> 16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11 >> 17:42:06 INFO cluster.YarnClientSchedulerBackend: Stopped >> 16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11 >> 17:42:06 INFO log.PerfLogger: </PERFLOG method=serializePlan >> start=1452505326849 end=1452505326874 duration=25 >> from=org.apache.hadoop.hive.ql.exec.Utilities> >> 16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11 >> 17:42:06 INFO log.PerfLogger: </PERFLOG method=SparkCreateTran.Reducer 2 >> start=1452505326824 end=1452505326936 duration=112 >> from=org.apache.hadoop.hive.ql.exec.spark.SparkPlanGenerator> >> 16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11 >> 17:42:06 INFO log.PerfLogger: </PERFLOG method=SparkBuildPlan >> start=1452505326273 end=1452505326936 duration=663 >> from=org.apache.hadoop.hive.ql.exec.spark.SparkPlanGenerator> >> 16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11 >> 17:42:06 INFO log.PerfLogger: <PERFLOG method=SparkBuildRDDGraph >> from=org.apache.hadoop.hive.ql.exec.spark.SparkPlan> >> 16/01/11 17:42:07 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11 >> 17:42:07 INFO log.PerfLogger: </PERFLOG method=SparkBuildRDDGraph >> start=1452505326936 end=1452505327000 duration=64 >> from=org.apache.hadoop.hive.ql.exec.spark.SparkPlan> >> 16/01/11 17:42:07 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11 >> 17:42:07 INFO client.RemoteDriver: Failed to run job >> b5a8886e-8229-4773-b943-b3c6f3c710bf >> 16/01/11 17:42:07 [stderr-redir-1]: INFO client.SparkClientImpl: >> java.lang.IllegalStateException: Cannot call methods on a stopped >> SparkContext >> 16/01/11 17:42:07 [stderr-redir-1]: INFO client.SparkClientImpl: at >> org.apache.spark.SparkContext.org >> <http://org.apache.spark.sparkcontext.org/>$apache$spark$SparkContext$$assertNotStopped(SparkContext.scala:104) >> 16/01/11 17:42:07 [stderr-redir-1]: INFO client.SparkClientImpl: at >> org.apache.spark.SparkContext.submitJob(SparkContext.scala:1981) >> 16/01/11 17:42:07 [stderr-redir-1]: INFO client.SparkClientImpl: at >> org.apache.spark.rdd.AsyncRDDActions$$anonfun$foreachAsync$1.apply(AsyncRDDActions.scala:118) >> 16/01/11 17:42:07 [stderr-redir-1]: INFO client.SparkClientImpl: at >> org.apache.spark.rdd.AsyncRDDActions$$anonfun$foreachAsync$1.apply(AsyncRDDActions.scala:116) >> 16/01/11 17:42:07 [stderr-redir-1]: INFO client.SparkClientImpl: at >> org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:147) >> 16/01/11 17:42:07 [stderr-redir-1]: INFO client.SparkClientImpl: at >> org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:108) >> 16/01/11 17:42:07 [stderr-redir-1]: INFO client.SparkClientImpl: at >> org.apache.spark.rdd.RDD.withScope(RDD.scala:310) >> 16/01/11 17:42:07 [stderr-redir-1]: INFO client.SparkClientImpl: at >> org.apache.spark.rdd.AsyncRDDActions.foreachAsync(AsyncRDDActions.scala:116) >> 16/01/11 17:42:07 [stderr-redir-1]: INFO client.SparkClientImpl: at >> org.apache.spark.api.java.JavaRDDLike$class.foreachAsync(JavaRDDLike.scala:690) >> 16/01/11 17:42:07 [stderr-redir-1]: INFO client.SparkClientImpl: at >> org.apache.spark.api.java.AbstractJavaRDDLike.foreachAsync(JavaRDDLike.scala:47) >> 16/01/11 17:42:07 [stderr-redir-1]: INFO client.SparkClientImpl: at >> org.apache.hadoop.hive.ql.exec.spark.RemoteHiveSparkClient$JobStatusJob.call(RemoteHiveSparkClient.java:257) >> 16/01/11 17:42:07 [stderr-redir-1]: INFO client.SparkClientImpl: at >> org.apache.hive.spark.client.RemoteDriver$JobWrapper.call(RemoteDriver.java:366) >> 16/01/11 17:42:07 [stderr-redir-1]: INFO client.SparkClientImpl: at >> org.apache.hive.spark.client.RemoteDriver$JobWrapper.call(RemoteDriver.java:335) >> 16/01/11 17:42:07 [stderr-redir-1]: INFO client.SparkClientImpl: at >> java.util.concurrent.FutureTask.run(FutureTask.java:262) >> 16/01/11 17:42:07 [stderr-redir-1]: INFO client.SparkClientImpl: at >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >> 16/01/11 17:42:07 [stderr-redir-1]: INFO client.SparkClientImpl: at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >> 16/01/11 17:42:07 [stderr-redir-1]: INFO client.SparkClientImpl: at >> java.lang.Thread.run(Thread.java:745) >> 16/01/11 17:42:07 [RPC-Handler-3]: INFO client.SparkClientImpl: Received >> result for b5a8886e-8229-4773-b943-b3c6f3c710bf >> Status: Failed >> 16/01/11 17:42:07 [main]: ERROR status.SparkJobMonitor: Status: Failed >> >> >> >> >> At 2016-01-11 17:17:51, "Sofia" <sofia.panagiot...@taiger.com >> <mailto:sofia.panagiot...@taiger.com>> wrote: >> Check log files to see the exact error. Alternatively run hive like this, to >> get the exact error >> >> hive --hiveconf hive.root.logger=INFO,console >> >> There is a good chance that you are encountering problems between the Hive >> and Spark versions and installation. >> See >> https://cwiki.apache.org/confluence/display/Hive/Hive+on+Spark%3A+Getting+Started >> >> <https://cwiki.apache.org/confluence/display/Hive/Hive+on+Spark%3A+Getting+Started> >> >> >>> On 11 Jan 2016, at 08:47, Todd <bit1...@163.com <mailto:bit1...@163.com>> >>> wrote: >>> >>> Hive, >>> >>> I am trying out the Hive on Spark with hive 1.2.1 and spark 1.5.2. Could >>> someone help me on this? Thanks! >>> >>> Following are my steps: >>> 1. build spark 1.5.2 without Hive and Hive Thrift Server. At this point, I >>> can use it to submit application using spark-submit --master yarn-client >>> 2. And the built spark assembly jar into $HIVE_HOME/lib >>> 3. start hive and add the following parameters >>> hive > set spark.master=yarn-client >>> hive > set spark.executor.memory=512M >>> hive > set spark.driver.memory=512M >>> hive > set spark.executor.instances=1 >>> >>> 4. Then I run a simple query : select count(1) from t1; >>> >>> The job fails will following error: >>> >>> =============================================================================== >>> YARN executor launch context: >>> env: >>> CLASSPATH -> >>> {{PWD}}<CPS>{{PWD}}/__spark__.jar<CPS>$HADOOP_CONF_DIR<CPS>$HADOOP_COMMON_HOME/share/hadoop/common/*<CPS>$HADOOP_COMMON_HOME/share/hadoop/common/lib/*<CPS>$HADOOP_HDFS_HOME/share/hadoop/hdfs/*<CPS>$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*<CPS>$HADOOP_YARN_HOME/share/hadoop/yarn/*<CPS>$HADOOP_YARN_HOME/share/hadoop/yarn/lib/*<CPS>$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*<CPS>$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/* >>> SPARK_LOG_URL_STDERR -> >>> http://hadoop-Aspire-TC-606:8042/node/containerlogs/container_1452320323183_0007_01_000003/hadoop/stderr?start=-4096 >>> >>> <http://hadoop-aspire-tc-606:8042/node/containerlogs/container_1452320323183_0007_01_000003/hadoop/stderr?start=-4096> >>> SPARK_YARN_STAGING_DIR -> .sparkStaging/application_1452320323183_0007 >>> SPARK_YARN_CACHE_FILES_FILE_SIZES -> 142746538 >>> SPARK_USER -> hadoop >>> SPARK_YARN_CACHE_FILES_VISIBILITIES -> PRIVATE >>> SPARK_YARN_MODE -> true >>> SPARK_YARN_CACHE_FILES_TIME_STAMPS -> 1452496343550 >>> SPARK_LOG_URL_STDOUT -> >>> http://hadoop-Aspire-TC-606:8042/node/containerlogs/container_1452320323183_0007_01_000003/hadoop/stdout?start=-4096 >>> >>> <http://hadoop-aspire-tc-606:8042/node/containerlogs/container_1452320323183_0007_01_000003/hadoop/stdout?start=-4096> >>> SPARK_YARN_CACHE_FILES -> >>> hdfs://hadoop.bit.com:9000/user/hadoop/.sparkStaging/application_1452320323183_0007/spark-assembly-1.5.2-hadoop2.6.0.jar#__spark__.jar >>> >>> <hdfs://hadoop.bit.com:9000/user/hadoop/.sparkStaging/application_1452320323183_0007/spark-assembly-1.5.2-hadoop2.6.0.jar#__spark__.jar> >>> >>> command: >>> {{JAVA_HOME}}/bin/java -server -XX:OnOutOfMemoryError='kill %p' >>> -Xms512m -Xmx512m >>> '-Dhive.spark.log.dir=/home/hadoop/software/bigdata/spark-1.5.2-bin-hadoop2.6.0-withouthive/logs/' >>> -Djava.io.tmpdir={{PWD}}/tmp '-Dspark.driver.port=43675' >>> -Dspark.yarn.app.container.log.dir=<LOG_DIR> >>> org.apache.spark.executor.CoarseGrainedExecutorBackend --driver-url >>> akka.tcp://sparkDriver@10.12.167.42:43675/user/CoarseGrainedScheduler >>> <akka.tcp://sparkDriver@10.12.167.42:43675/user/CoarseGrainedScheduler> >>> --executor-id 2 --hostname hadoop-Aspire-TC-606 --cores 1 --app-id >>> application_1452320323183_0007 --user-class-path file:$PWD/__app__.jar 1> >>> <LOG_DIR>/stdout 2> <LOG_DIR>/stderr >>> =============================================================================== >>> >>> 16/01/11 15:12:37 INFO impl.ContainerManagementProtocolProxy: Opening proxy >>> : hadoop-Aspire-TC-606:50804 >>> 16/01/11 15:12:40 INFO yarn.YarnAllocator: Completed container >>> container_1452320323183_0007_01_000003 (state: COMPLETE, exit status: 1) >>> 16/01/11 15:12:40 INFO yarn.YarnAllocator: Container marked as failed: >>> container_1452320323183_0007_01_000003. Exit status: 1. Diagnostics: >>> Exception from container-launch. >>> Container id: container_1452320323183_0007_01_000003 >>> Exit code: 1 >>> Stack trace: ExitCodeException exitCode=1: >>> at org.apache.hadoop.util.Shell.runCommand(Shell.java:538) >>> at org.apache.hadoop.util.Shell.run(Shell.java:455) >>> at >>> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:715) >>> at >>> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:211) >>> at >>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302) >>> at >>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82) >>> at java.util.concurrent.FutureTask.run(FutureTask.java:262) >>> at >>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >>> at >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >>> at java.lang.Thread.run(Thread.java:745) >>> >>> >>> Container exited with a non-zero exit code 1 >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >> >
