[
https://issues.apache.org/jira/browse/SPARK-19055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Apache Spark reassigned SPARK-19055:
------------------------------------
Assignee: Apache Spark
> SparkSession initialization will be associated with invalid SparkContext when
> new SparkContext is created to replace stopped SparkContext
> -----------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-19055
> URL: https://issues.apache.org/jira/browse/SPARK-19055
> Project: Spark
> Issue Type: Bug
> Components: PySpark, SQL
> Reporter: Liang-Chi Hsieh
> Assignee: Apache Spark
>
> In SparkSession initialization, we store created the instance of SparkSession
> into a class variable _instantiatedContext. Next time we can use
> SparkSession.builder.getOrCreate() to retrieve the existing SparkSession
> instance.
> However, when the active SparkContext is stopped and we create another new
> SparkContext to use, the existing SparkSession is still associated with the
> stopped SparkContext. So the operations with this existing SparkSession will
> be failed.
> We need to detect such case in SparkSession and renew the class variable
> _instantiatedContext if needed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
