In cluster settings if you don't explicitly call sc.stop() your application may hang. Like closing files, network connections, etc, when you're done with them, it's a good idea to call sc.stop(), which lets the spark master know that your application is finished consuming resources.
On Fri, Oct 31, 2014 at 10:13 AM, Daniel Siegmann <[email protected]> wrote: > It is used to shut down the context when you're done with it, but if > you're using a context for the lifetime of your application I don't think > it matters. > > I use this in my unit tests, because they start up local contexts and you > can't have multiple local contexts open so each test must stop its context > when it's done. > > On Fri, Oct 31, 2014 at 11:12 AM, ll <[email protected]> wrote: > >> what is it for? when do we call it? >> >> thanks! >> >> >> >> -- >> View this message in context: >> http://apache-spark-user-list.1001560.n3.nabble.com/SparkContext-stop-tp17826.html >> Sent from the Apache Spark User List mailing list archive at Nabble.com. >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: [email protected] >> For additional commands, e-mail: [email protected] >> >> > > > -- > Daniel Siegmann, Software Developer > Velos > Accelerating Machine Learning > > 440 NINTH AVENUE, 11TH FLOOR, NEW YORK, NY 10001 > E: [email protected] W: www.velos.io >
