I'm trying to log Tasks to understand physical plan and to visualize which
RDD's which partition is currently computed from which creation site along
with other information. I want to charge the TaskRunner to do this before
actually invoking runTask() on Task and again just before giving the Task
t
That's correct. What's the reason this information is needed?
-Sandy
On Tue, Mar 24, 2015 at 11:41 AM, Zoltán Zvara
wrote:
> Thank you for your response!
>
> I guess the (Spark)AM, who gives the container leash to the NM (along with
> the executor JAR and command to run) must know how many CPU
Thank you for your response!
I guess the (Spark)AM, who gives the container leash to the NM (along with
the executor JAR and command to run) must know how many CPU or RAM that
container capped, isolated at. There must be a resource vector along the
encrypted container leash if I'm right that descr
Hi Zoltan,
If running on YARN, the YARN NodeManager starts executors. I don't think
there's a 100% precise way for the Spark executor way to know how many
resources are allotted to it. It can come close by looking at the Spark
configuration options used to request it (spark.executor.memory and
s
Let's say I'm an Executor instance in a Spark system. Who started me and
where, when I run on a worker node supervised by (a) Mesos, (b) YARN? I
suppose I'm the only one Executor on a worker node for a given framework
scheduler (driver). If I'm an Executor instance, who is the closest object
to me
