Also - I double checked - we're setting the master to "yarn-cluster"
-----Original Message----- From: "Tathagata Das" <t...@databricks.com> Sent: 9/23/2015 2:38 PM To: "Bryan" <bryan.jeff...@gmail.com> Cc: "user" <user@spark.apache.org>; "Hari Shreedharan" <hshreedha...@cloudera.com> Subject: Re: Yarn Shutting Down Spark Processing CC;ing Hari who may have a better sense of whats going on. ---------- Forwarded message ---------- From: Bryan <bryan.jeff...@gmail.com> Date: Wed, Sep 23, 2015 at 3:43 AM Subject: RE: Yarn Shutting Down Spark Processing To: Tathagata Das <t...@databricks.com> Cc: user <user@spark.apache.org> Tathagata, Simple batch jobs do work. The cluster has a good set of resources and a limited input volume on the given Kafka topic. The job works on the small 3-node standalone-configured cluster I have setup for test. Regards, Bryan Jeffrey From: Tathagata Das Sent: 9/23/2015 2:46 AM To: Bryan Jeffrey Cc: user Subject: Re: Yarn Shutting Down Spark Processing Does your simple Spark batch jobs work in the same YARN setup? May be YARN is not able to provide resources that you are asking for. On Tue, Sep 22, 2015 at 5:49 PM, Bryan Jeffrey <bryan.jeff...@gmail.com> wrote: Hello. I have a Spark streaming job running on a cluster managed by Yarn. The spark streaming job starts and receives data from Kafka. It is processing well and then after several seconds I see the following error: 15/09/22 14:53:49 ERROR yarn.ApplicationMaster: SparkContext did not initialize after waiting for 100000 ms. Please check earlier log output for errors. Failing the application. 15/09/22 14:53:49 INFO yarn.ApplicationMaster: Final app status: FAILED, exitCode: 13, (reason: Timed out waiting for SparkContext.) The spark process is then (obviously) shut down by Yarn. What do I need to change to allow Yarn to initialize Spark streaming (vs. batch) jobs? Thank you, Bryan Jeffrey
