Ah, python. How about SparkContext._active_spark_context then? On Tue, Aug 7, 2018 at 5:34 PM Andrew Melo <andrew.m...@gmail.com> wrote:
> Hi Sean, > > On Tue, Aug 7, 2018 at 5:16 PM, Sean Owen <sro...@gmail.com> wrote: > > Is SparkSession.getActiveSession what you're looking for? > > Perhaps -- though there's not a corresponding python function, and I'm > not exactly sure how to call the scala getActiveSession without first > instantiating the python version and causing a JVM to start. > > Is there an easy way to call getActiveSession that doesn't start a JVM? > > Cheers > Andrew > > > > > On Tue, Aug 7, 2018 at 5:11 PM Andrew Melo <andrew.m...@gmail.com> > wrote: > >> > >> Hello, > >> > >> One pain point with various Jupyter extensions [1][2] that provide > >> visual feedback about running spark processes is the lack of a public > >> API to introspect the web URL. The notebook server needs to know the > >> URL to find information about the current SparkContext. > >> > >> Simply looking for "localhost:4040" works most of the time, but fails > >> if multiple spark notebooks are being run on the same host -- spark > >> increments the port for each new context, leading to confusion when > >> the notebooks are trying to probe the web interface for information. > >> > >> I'd like to implement an analog to SparkContext.getOrCreate(), perhaps > >> called "getIfExists()" that returns the current singleton if it > >> exists, or None otherwise. The Jupyter code would then be able to use > >> this entrypoint to query Spark for an active Spark context, which it > >> could use to probe the web URL. > >> > >> It's a minor change, but this would be my first contribution to Spark, > >> and I want to make sure my plan was kosher before I implemented it. > >> > >> Thanks! > >> Andrew > >> > >> > >> > >> > >> > >> [1] https://krishnan-r.github.io/sparkmonitor/ > >> > >> [2] https://github.com/mozilla/jupyter-spark > >> > >> --------------------------------------------------------------------- > >> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org > >> > > >
