actually if you search the spark mail archives you will find many similar
topics. At this time, I just want to manage it by myself.
On Tuesday, August 12, 2014 8:46 PM, Stanley Shi wrote:
This seems a bug, right? It's not the user's responsibility to manage the
workers.
On Wed, Aug 13,
This seems a bug, right? It's not the user's responsibility to manage the
workers.
On Wed, Aug 13, 2014 at 11:28 AM, S. Zhou wrote:
> Sometimes workers are dead but spark context does not know it and still
> send jobs.
>
>
> On Tuesday, August 12, 2014 7:14 PM, Stanley Shi
> wrote:
>
>
> Why
Sometimes workers are dead but spark context does not know it and still send
jobs.
On Tuesday, August 12, 2014 7:14 PM, Stanley Shi wrote:
Why do you need to detect the worker status in the application? you application
generally don't need to know where it is executed.
On Wed, Aug 13, 2
I tried to access worker info from spark context but it seems spark context
does no expose such API. The reason of doing that is: it seems spark context
itself does not have logic to detect if its workers are in dead status. So I
like to add such logic by myself.
BTW, it seems spark web UI has
