Thanks for the help. I read that driver only send a function (task) to
executors and the executors apply this function to their local RDD
partitions.
Iacovos
On 1/9/20 5:03 PM, Wenchen Fan wrote:
RDD has a flag `storageLevel` which will be set by calling persist().
RDD will be serialized and sent to executors for running tasks. So
executors just look at RDD.storageLevel and store output in its block
manager when needed.
On Thu, Jan 9, 2020 at 5:53 PM Jack Kolokasis <koloka...@ics.forth.gr
<mailto:koloka...@ics.forth.gr>> wrote:
Hello all,
I want to find when a Task that is sended by Driver to executor
contains
a call to function persist(). I am trying to read the submitted
function
that driver send to executor but I could not find any call to
persist()
method. Do you know how executor understand which RDDs needed to
be persist?
Thanks,
Iacovos Kolokasis
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
<mailto:dev-unsubscr...@spark.apache.org>
