Interesting and useful. I would love to see your changes for reference. Have you submitted this feature back to the SPARK community? I think this would be useful for a lot of interactive spark use-cases in shared clusters.
Also I have a question regarding your experience maintaining your own SPARK fork. We have also been forced to maintain a very minor spark fork to work around some bugs in our environment, but our internal goal is to try to get back to the main apache branch or a vendor fork. Have you found keeping up with SPARK (which is changing lightning fast – pun intended) challenging? Thanks, Shaun Litt From: Rajat Gupta [mailto:rajat.iit...@gmail.com] Sent: Wednesday, July 01, 2015 2:41 AM To: users@zeppelin.incubator.apache.org Subject: Re: Zeppelin Interpreter end We have implemented auto shutdown of spark application by adding an idle timeout setting directly in spark. We set the value of this setting in the zeppelin interpreter UI page and default it to 60 mins. For this fix we also added an auto lazy restart of interpreter in zeppelin if interpreter has shutdown because of inactivity. If you want I can send you patches for this both for spark and zeppelin. I did not submit the patch in zeppelin for this earlier because this makes sense only if spark has idle timeout in open source. Dynamic scaling of spark applications works fine on zeppelin with appropriate spark settings on interpreter page. Zeppelin just maintains handle to a spark context and that spark context is free to do anything afaik. We at Qubole are using it actively. Rajat On Tue, Jun 30, 2015 at 4:53 AM, Litt, Shaun <sl...@conversantmedia.com<mailto:sl...@conversantmedia.com>> wrote: Hi, I am new to zeppelin and just got it configured to run in my YARN cluster, but I was wondering if there is a configuration or even a hard setting that shuts down interpreters after in-activity. It seems like the interpreter (and it’s yarn consumption) hang out indefinitely, ideally there would be a clean way (like logout or a shutdown button within the notebook) to shutdown these interpreters, but additionally there should be a way for an admin of zeppelin to impose and idle timeout. As a note to the scope of this, is dynamic yarn resource allocation configured (such that once a paragraph finishes it can release vcores)? Thanks, Shaun This email and any files included with it may contain privileged, proprietary and/or confidential information that is for the sole use of the intended recipient(s). Any disclosure, copying, distribution, posting, or use of the information contained in or attached to this email is prohibited unless permitted by the sender. If you have received this email in error, please immediately notify the sender via return email, telephone, or fax and destroy this original transmission and its included files without reading or saving it in any manner. Thank you. This email and any files included with it may contain privileged, proprietary and/or confidential information that is for the sole use of the intended recipient(s). Any disclosure, copying, distribution, posting, or use of the information contained in or attached to this email is prohibited unless permitted by the sender. If you have received this email in error, please immediately notify the sender via return email, telephone, or fax and destroy this original transmission and its included files without reading or saving it in any manner. Thank you.
