[
https://issues.apache.org/jira/browse/SPARK-13747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16000788#comment-16000788
]
Mousa HAMAD commented on SPARK-13747:
-------------------------------------
Thanks to [~dnaumenko], we also managed to avoid running into this issue by
switching akka’s default executor to “thread-pool-executor”. We are also using
akka a lot in our system that is creating Spark’s jobs.
What I need to make sure of that, is this issue really a Spark issue or a
fork-join framework issue? As far as I understood, a thread can be reused by
its (or another) process before completely finishing its work to perform
another work which would pollute its locals. If this is the case then the
solution proposed in the pull request
https://github.com/apache/spark/pull/17763 won’t resolve this issue as it
simply, if I fully got the proposed solution, prevents timeout exceptions from
being thrown when calling "Await.ready".
> Concurrent execution in SQL doesn't work with Scala ForkJoinPool
> ----------------------------------------------------------------
>
> Key: SPARK-13747
> URL: https://issues.apache.org/jira/browse/SPARK-13747
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 2.0.0, 2.0.1
> Reporter: Shixiong Zhu
> Assignee: Shixiong Zhu
>
> Run the following codes may fail
> {code}
> (1 to 100).par.foreach { _ =>
> println(sc.parallelize(1 to 5).map { i => (i, i) }.toDF("a", "b").count())
> }
> java.lang.IllegalArgumentException: spark.sql.execution.id is already set
> at
> org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:87)
>
> at
> org.apache.spark.sql.DataFrame.withNewExecutionId(DataFrame.scala:1904)
> at org.apache.spark.sql.DataFrame.collect(DataFrame.scala:1385)
> {code}
> This is because SparkContext.runJob can be suspended when using a
> ForkJoinPool (e.g.,scala.concurrent.ExecutionContext.Implicits.global) as it
> calls Await.ready (introduced by https://github.com/apache/spark/pull/9264).
> So when SparkContext.runJob is suspended, ForkJoinPool will run another task
> in the same thread, however, the local properties has been polluted.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
