Thanks Josh and Marcelo! It now works!
BTW, just wondering, is there any perf difference between running spark in
standalone mode and under yarn? The only goal that I created this cluster is to
run spark jobs. So I can set up spark in standalone mode if it runs slow in
yarn.
best.
From: Josh Rosen <[email protected]>
To: Marcelo Vanzin <[email protected]>
Cc: freedafeng <[email protected]>; "[email protected]"
<[email protected]>
Sent: Wednesday, January 14, 2015 3:20 PM
Subject: Re: how to run python app in yarn?
There's an open PR for supporting yarn-cluster mode in PySpark:
https://github.com/apache/spark/pull/3976 (currently blocked on reviewer
attention / time)
On Wed, Jan 14, 2015 at 3:16 PM, Marcelo Vanzin <[email protected]> wrote:
As the error message says...
On Wed, Jan 14, 2015 at 3:14 PM, freedafeng <[email protected]> wrote:
> Error: Cluster deploy mode is currently not supported for python
> applications.
Use "yarn-client" instead of "yarn-cluster" for pyspark apps.
--
Marcelo
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
