[
https://issues.apache.org/jira/browse/SPARK-19476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15942380#comment-15942380
]
Gal Topper commented on SPARK-19476:
------------------------------------
BTW, my workaround was quite painful: I created a single-threaded producer that
used the original executor thread to feed the data into the async stream.
Here's the code, in case anyone stumbles on this and is looking for a way out:
{code:title=SingleThreadedPublisher.scala|borderStyle=solid}
import org.reactivestreams.{Publisher, Subscriber, Subscription}
/** A reactive streams publisher to work around SPARK-19476 by only using
Spark's iterator from the original thread. */
private class SingleThreadedPublisher[T] extends Publisher[T] {
private var cancelled = false
private var demand = 0L
private var subscriber: Subscriber[_ >: T] = _
private val waitForDemandObject = new Object
override def subscribe(s: Subscriber[_ >: T]): Unit = {
this.subscriber = s
val subscription = new Subscription {
override def cancel(): Unit = waitForDemandObject.synchronized {
cancelled = true
waitForDemandObject.notify()
}
override def request(n: Long): Unit = {
waitForDemandObject.synchronized {
demand += n
waitForDemandObject.notify()
}
}
}
s.onSubscribe(subscription)
}
private def produce(element: T): Unit = {
demand -= 1
subscriber.onNext(element)
}
def push(iterator: Iterator[T]): Unit = {
iterator.takeWhile(_ => !cancelled).foreach { element =>
waitForDemandObject.synchronized {
if (demand > 0L) {
produce(element)
} else {
waitForDemandObject.wait()
produce(element)
}
}
}
if (!cancelled) {
subscriber.onComplete()
}
}
}
{code}
> Running threads in Spark DataFrame foreachPartition() causes
> NullPointerException
> ---------------------------------------------------------------------------------
>
> Key: SPARK-19476
> URL: https://issues.apache.org/jira/browse/SPARK-19476
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 1.6.0, 1.6.1, 1.6.2, 1.6.3, 2.0.0, 2.0.1, 2.0.2, 2.1.0
> Reporter: Gal Topper
>
> First reported on [Stack
> overflow|http://stackoverflow.com/questions/41674069/running-threads-in-spark-dataframe-foreachpartition].
> I use multiple threads inside foreachPartition(), which works great for me
> except for when the underlying iterator is TungstenAggregationIterator. Here
> is a minimal code snippet to reproduce:
> {code:title=Reproduce.scala|borderStyle=solid}
> import scala.concurrent.ExecutionContext.Implicits.global
> import scala.concurrent.duration.Duration
> import scala.concurrent.{Await, Future}
> import org.apache.spark.SparkContext
> import org.apache.spark.sql.SQLContext
> object Reproduce extends App {
> val sc = new SparkContext("local", "reproduce")
> val sqlContext = new SQLContext(sc)
> import sqlContext.implicits._
> val df = sc.parallelize(Seq(1)).toDF("number").groupBy("number").count()
> df.foreachPartition { iterator =>
> val f = Future(iterator.toVector)
> Await.result(f, Duration.Inf)
> }
> }
> {code}
> When I run this, I get:
> {noformat}
> java.lang.NullPointerException
> at
> org.apache.spark.sql.execution.aggregate.TungstenAggregationIterator.next(TungstenAggregationIterator.scala:751)
> at
> org.apache.spark.sql.execution.aggregate.TungstenAggregationIterator.next(TungstenAggregationIterator.scala:84)
> at scala.collection.Iterator$$anon$11.next(Iterator.scala:409)
> at scala.collection.Iterator$class.foreach(Iterator.scala:893)
> at scala.collection.AbstractIterator.foreach(Iterator.scala:1336)
> {noformat}
> I believe I actually understand why this happens -
> TungstenAggregationIterator uses a ThreadLocal variable that returns null
> when called from a thread other than the original thread that got the
> iterator from Spark. From examining the code, this does not appear to differ
> between recent Spark versions.
> However, this limitation is specific to TungstenAggregationIterator, and not
> documented, as far as I'm aware.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
