Hi, My little understanding of Python-Spark bridge is that at some point the python code communicates over the wire with Spark's backbone that includes PythonRDD [1].
When the CallSite can't be computed, it's null:-1 to denote "nothing could be referred to". [1] https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/api/python/PythonRDD.scala Pozdrawiam, Jacek Laskowski ---- https://medium.com/@jaceklaskowski/ Mastering Apache Spark http://bit.ly/mastering-apache-spark Follow me at https://twitter.com/jaceklaskowski On Thu, Jul 21, 2016 at 8:36 PM, C. Josephson <cjos...@uhana.io> wrote: >> It's called a CallSite that shows where the line comes from. You can see >> the code yourself given the python file and the line number. > > > But that's what I don't understand. Which python file? We spark submit one > file called ctr_parsing.py, but it only has 150 lines. So what is > MapPartitions at PythonRDD.scala:374 referring to? ctr_parsing.py imports a > number of support functions we wrote, but how do we know which python file > to look at? > > Furthermore, what on earth is null:-1 referring to? --------------------------------------------------------------------- To unsubscribe e-mail: user-unsubscr...@spark.apache.org
