Does your simple Spark batch jobs work in the same YARN setup? May be YARN is not able to provide resources that you are asking for.
On Tue, Sep 22, 2015 at 5:49 PM, Bryan Jeffrey <bryan.jeff...@gmail.com> wrote: > Hello. > > I have a Spark streaming job running on a cluster managed by Yarn. The > spark streaming job starts and receives data from Kafka. It is processing > well and then after several seconds I see the following error: > > 15/09/22 14:53:49 ERROR yarn.ApplicationMaster: SparkContext did not > initialize after waiting for 100000 ms. Please check earlier log output for > errors. Failing the application. > 15/09/22 14:53:49 INFO yarn.ApplicationMaster: Final app status: FAILED, > exitCode: 13, (reason: Timed out waiting for SparkContext.) > > The spark process is then (obviously) shut down by Yarn. > > What do I need to change to allow Yarn to initialize Spark streaming (vs. > batch) jobs? > > Thank you, > > Bryan Jeffrey >
