This seems a bug, right? It's not the user's responsibility to manage the workers.
On Wed, Aug 13, 2014 at 11:28 AM, S. Zhou <myx...@yahoo.com> wrote: > Sometimes workers are dead but spark context does not know it and still > send jobs. > > > On Tuesday, August 12, 2014 7:14 PM, Stanley Shi <s...@pivotal.io> > wrote: > > > Why do you need to detect the worker status in the application? you > application generally don't need to know where it is executed. > > > On Wed, Aug 13, 2014 at 7:39 AM, S. Zhou <myx...@yahoo.com.invalid> wrote: > > I tried to access worker info from spark context but it seems spark > context does no expose such API. The reason of doing that is: it seems > spark context itself does not have logic to detect if its workers are in > dead status. So I like to add such logic by myself. > > BTW, it seems spark web UI has some logic of detecting dead workers. But > all relevant classes are declared as private for the spark package scope. > > Please let me know how to solve this issue (or if there is an alternative > way to achieve the same purpose) > > Thanks > > > > > -- > Regards, > *Stanley Shi,* > > > > -- Regards, *Stanley Shi,*
