I actually had a similar use case a while ago, but not entirely the same. In my use case, Spark is already up, but I want to make sure all existing (and new) executors run some specific code. Can we update the API to support that? I think that's doable if we split the design into two: one is the ability to do what I just mentioned, and second is the ability to register via config class when Spark starts to run the code.
On Thu, Aug 30, 2018 at 11:01 PM Felix Cheung <felixcheun...@hotmail.com> wrote: > +1 > ------------------------------ > *From:* Mridul Muralidharan <mri...@gmail.com> > *Sent:* Wednesday, August 29, 2018 1:27:27 PM > *To:* dev@spark.apache.org > *Subject:* Re: SPIP: Executor Plugin (SPARK-24918) > > +1 > I left a couple of comments in NiharS's PR, but this is very useful to > have in spark ! > > Regards, > Mridul > On Fri, Aug 3, 2018 at 10:00 AM Imran Rashid > <iras...@cloudera.com.invalid> wrote: > > > > I'd like to propose adding a plugin api for Executors, primarily for > instrumentation and debugging ( > https://issues.apache.org/jira/browse/SPARK-24918). The changes are > small, but as its adding a new api, it might be spip-worthy. I mentioned > it as well in a recent email I sent about memory monitoring > > > > The spip proposal is here (and attached to the jira as well): > https://docs.google.com/document/d/1a20gHGMyRbCM8aicvq4LhWfQmoA5cbHBQtyqIA2hgtc/edit?usp=sharing > > > > There are already some comments on the jira and pr, and I hope to get > more thoughts and opinions on it. > > > > thanks, > > Imran > > --------------------------------------------------------------------- > To unsubscribe e-mail: dev-unsubscr...@spark.apache.org > >
