Thanks Moon!
On Fri, Nov 4, 2016 at 1:24 PM moon soo Lee <m...@apache.org> wrote: > Hi Aish, > > Deprecating %dep was plan before. %dep loads library into thread context > classloader after JVM process creation. However, in few places, Spark tries > to find class from System classloader, not from classloader of thread > context. That was reason we introduced new dependency loading mechanism > (through interpreter GUI) which set library into the JVM class path on > process creation, and deprecating %dep. > > But since then, we've got multiple feedbacks from user that they likes > %dep approaches for some reasons. > > - %dep is self-documenting library requirement in the notebook. > - %dep allows per Note (and possible per User) library loading. > > And i agree. So, It make sense to keep %dep unless we have other > alternatives. > > Thanks, > moon > > On Fri, Nov 4, 2016 at 12:53 PM Aish Fenton <afen...@netflix.com> wrote: > > Hi all, > Can someone clear up if %dep is going to be deprecated? From the docs and > warning messages it sounds like that's still the plan. But from an in > person conversation I've had with Moon, it sounded like maybe not (although > this was a while back)? > > If it's depreciated, that'll leave a big gap for us. Our typical workflow > involves us exchanging notebooks (export/import of the json) fairly > regularly. And every notebook has different library dependencies. Being > able to specify these inline, in the notebook, is really useful. > > I remember the concern with %dep was with adding jars dynamically? If this > is still true, then maybe a compromise would be to keep a way of > documenting the jar dependencies per notebook, but when the user loads that > notebook prompt them with "Jar X is missing, click yes to add to your Spark > interpreter dependencies". So have it function more like an automatic way > to fill in the GUI dependencies settings, but at least it keeps the > self-documenting functionality too. > > Aish > >
