We have implemented auto shutdown of spark application by adding an idle timeout setting directly in spark. We set the value of this setting in the zeppelin interpreter UI page and default it to 60 mins. For this fix we also added an auto lazy restart of interpreter in zeppelin if interpreter has shutdown because of inactivity. If you want I can send you patches for this both for spark and zeppelin. I did not submit the patch in zeppelin for this earlier because this makes sense only if spark has idle timeout in open source.
Dynamic scaling of spark applications works fine on zeppelin with appropriate spark settings on interpreter page. Zeppelin just maintains handle to a spark context and that spark context is free to do anything afaik. We at Qubole are using it actively. Rajat On Tue, Jun 30, 2015 at 4:53 AM, Litt, Shaun <sl...@conversantmedia.com> wrote: > Hi, I am new to zeppelin and just got it configured to run in my YARN > cluster, but I was wondering if there is a configuration or even a hard > setting that shuts down interpreters after in-activity. It seems like the > interpreter (and it’s yarn consumption) hang out indefinitely, ideally > there would be a clean way (like logout or a shutdown button within the > notebook) to shutdown these interpreters, but additionally there should be > a way for an admin of zeppelin to impose and idle timeout. As a note to > the scope of this, is dynamic yarn resource allocation configured (such > that once a paragraph finishes it can release vcores)? > > > > Thanks, > > *Shaun* > > > > This email and any files included with it may contain privileged, > proprietary and/or confidential information that is for the sole use > of the intended recipient(s). Any disclosure, copying, distribution, > posting, or use of the information contained in or attached to this > email is prohibited unless permitted by the sender. If you have > received this email in error, please immediately notify the sender > via return email, telephone, or fax and destroy this original transmission > and its included files without reading or saving it in any manner. > Thank you. >
