Thanks for the reply, I have doubt as to which path to set for YARN_CONF_DIR
My /etc/hadoop folder has the following sub folders conf conf.cloudera.hdfs conf.cloudera.mapreduce conf.cloudera.yarn and both conf and conf.cloudera.yarn folders have yarn-site.xml. As of now I set the variable as export YARN_CONF_DIR=/etc/hadoop/conf But no change in the error .. only driver is added no executors for it, do i need to make any change in conf files or what is the correct path for variable. On Wed, Sep 24, 2014 at 11:18 PM, Greg Hill <greg.h...@rackspace.com> wrote: > Do you have YARN_CONF_DIR set in your environment to point Spark to > where your yarn configs are? > > Greg > > From: Raghuveer Chanda <raghuveer.cha...@gmail.com> > Date: Wednesday, September 24, 2014 12:25 PM > To: "u...@spark.incubator.apache.org" <u...@spark.incubator.apache.org> > Subject: Spark with YARN > > Hi, > > I'm new to spark and facing problem with running a job in cluster using > YARN. > > Initially i ran jobs using spark master as --master spark://dml2:7077 > and it is running fine on 3 workers. > > But now im shifting to YARN, so installed YARN in cloud era on 3 node > cluster and changed the master to yarn-cluster but it is not working I > attached the screenshots of UI which are not progressing and just hanging > on. > > Output on terminal : > > This error is repeating > > ./spark-submit --class "class-name" --master yarn-cluster > --num-executors 3 --executor-cores 3 jar-with-dependencies.jar > > > Do i need to configure YARN or why it is not getting all the workers .. > please help ... > > > 14/09/24 22:44:21 INFO yarn.Client: Application report from ASM: > application identifier: application_1411578463780_0001 > appId: 1 > clientToAMToken: null > appDiagnostics: > appMasterHost: dml3 > appQueue: root.chanda > appMasterRpcPort: 0 > appStartTime: 1411578513545 > yarnAppState: RUNNING > distributedFinalState: UNDEFINED > appTrackingUrl: http://dml2:8088/proxy/application_1411578463780_0001/ > appUser: chanda > 14/09/24 22:44:22 INFO yarn.Client: Application report from ASM: > application identifier: application_1411578463780_0001 > appId: 1 > clientToAMToken: null > appDiagnostics: > appMasterHost: dml3 > appQueue: root.chanda > appMasterRpcPort: 0 > appStartTime: 1411578513545 > yarnAppState: RUNNING > distributedFinalState: UNDEFINED > appTrackingUrl: http://dml2:8088/proxy/application_1411578463780_0001/ > > > > > -- > Regards, > Raghuveer Chanda > 4th year Undergraduate Student > Computer Science and Engineering > IIT Kharagpur > > -- Regards, Raghuveer Chanda 4th year Undergraduate Student Computer Science and Engineering IIT Kharagpur
