Thanks Mich. Great explanation
On Saturday, 11 June 2016, 22:35, Mich Talebzadeh
wrote:
Hi Gavin,
I believe in standalone mode a simple cluster manager is included with Spark
that makes it easyto set up a cluster.It does not rely on YARN or Mesos.
In summary this is from my notes:
Hi Ashok
Your points:
"
I know I can start spark-shell by launching the shell itself
spark-shell
Now I know that in standalone mode I can also connect to master
spark-shell --master spark://:7077
My point is what are the differences between these two start-up modes for
spark-shell? If I start
Sorry I have a typo.
Which means spark does not use yarn or mesos in standalone mode...
> On Jun 11, 2016, at 14:35, Mich Talebzadeh wrote:
>
> Hi Gavin,
>
> I believe in standalone mode a simple cluster manager is included with Spark
> that makes it easy to set up a cluster. It does not r
Hi Gavin,
I believe in standalone mode a simple cluster manager is included with
Spark that makes it easy to set up a cluster. It does not rely on YARN or
Mesos.
In summary this is from my notes:
-
Spark Local - Spark runs on the local host. This is the simplest set up
and best suited
The standalone mode is against Yarn mode or Mesos mode, which means spark
uses Yarn or Mesos as cluster managements.
Local mode is actually a standalone mode which everything runs on the
single local machine instead of remote clusters.
That is my understanding.
On Sat, Jun 11, 2016 at 12:40 PM,
Thank you for grateful
I know I can start spark-shell by launching the shell itself
spark-shell
Now I know that in standalone mode I can also connect to master
spark-shell --master spark://:7077
My point is what are the differences between these two start-up modes for
spark-shell? If I start sp
Hi Ashok,
In local mode all the processes run inside a single jvm, whereas in
standalone mode we have separate master and worker processes running in
their own jvms.
To quickly test your code from within your IDE you could probable use the
local mode. However, to get a real feel of how Spark oper
