Re: Add hot-deploy capability in Spark Shell

2016-06-06 Thread Kai Chen
I don't. The hot-deploy shouldn't happen while there is a job running. At least in the REPL it won't make much sense. It's a development-only feature to shorten the iterative coding cycle. In production environment, this is not enabled ... though there might be situations where it would be desi

Re: Add hot-deploy capability in Spark Shell

2016-06-06 Thread Reynold Xin
Thanks for the email. How do you deal with in-memory state that reference the classes? This can happen in both streaming and caching in RDD and temporary view creation in SQL. On Mon, Jun 6, 2016 at 3:40 PM, S. Kai Chen wrote: > Hi, > > We use spark-shell heavily for ad-hoc data analysis as well