Artem Aliev created SPARK-19362:
-----------------------------------
Summary: master UI kill link stops spark context but leave it
active
Key: SPARK-19362
URL: https://issues.apache.org/jira/browse/SPARK-19362
Project: Spark
Issue Type: Bug
Components: Spark Core
Affects Versions: 2.1.0, 2.0.2
Reporter: Artem Aliev
SparkGremlinComupter recreates internal SparkContext in case of error or if it
was killed. https://issues.apache.org/jira/browse/TINKERPOP-1271
By the way, If Spark application was killed from Master UI with (kill) link,
the SparkContext is not recreated.
It is marked as stopped but not removed from active context holder.
So there is no way to create new spark context, getOrCreate() method return old
stopped context.
The root of the problem is:
sc.stop() method is called from event-loop thread. It interrupt the thread, and
so it interrupt its own execution and do not clean active context.
You can use following example to reproduce the problem.
1. Start test application
2. kill application when it asked.
3, see error
{code}
import org.apache.spark.{SparkConf, SparkContext}
object StopTest {
def main(args: Array[String]) {
val sc = test
println("you have 15 sec to stop the context")
Thread.sleep(15000)
if(sc.isStopped) println ("SparkContext was stopped")
test
}
def test = {
println("Starting context")
val conf = new SparkConf().setAppName("Simple Spark Application")
val sc = new SparkContext(new SparkConf())
sc.parallelize(Seq(1,2,3)).count()
sc
}
}
{code}
Output:
{code}
SparkContext was stopped
Starting context
Exception in thread "main" org.apache.spark.SparkException: Only one
SparkContext may be running in this JVM (see SPARK-2243). To ignore this error,
set spark.driver.allowMultipleContexts = true. The currently running
SparkContext was created at:
org.apache.spark.SparkContext.<init>(SparkContext.scala:77)
StopTest$.test(StopTest.scala:14)
StopTest$.main(StopTest.scala:5)
StopTest.main(StopTest.scala)
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
java.lang.reflect.Method.invoke(Method.java:497)
org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:736)
org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:185)
org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:210)
org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:124)
org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
at
org.apache.spark.SparkContext$$anonfun$assertNoOtherContextIsRunning$2.apply(SparkContext.scala:2223)
at
org.apache.spark.SparkContext$$anonfun$assertNoOtherContextIsRunning$2.apply(SparkContext.scala:2219)
at scala.Option.foreach(Option.scala:257)
at
org.apache.spark.SparkContext$.assertNoOtherContextIsRunning(SparkContext.scala:2219)
at
org.apache.spark.SparkContext$.markPartiallyConstructed(SparkContext.scala:2292)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:86)
at StopTest$.test(StopTest.scala:14)
at StopTest$.main(StopTest.scala:9)
at StopTest.main(StopTest.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at
org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:736)
at
org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:185)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:210)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:124)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
