Should I be able to broadcast a fairly simple user-defined class? I'm having
no success in 1.6.0 (or 1.5.2):
$ cat test_spark.py
import pyspark
class SimpleClass:
def __init__(self):
self.val = 5
def get(self):
return self.val
def main():
sc = pyspark.SparkContext()
b = sc.broadcast(SimpleClass())
results = sc.parallelize([ x for x in range(10) ]).map(lambda x: x +
b.value.get()).collect()
if __name__ == '__main__':
main()
$ spark-submit --master local[1] test_spark.py
[snip]
File "/Users/ed/src/mrspark/examples/fortyler/test_spark.py", line 14, in
<lambda>
results = sc.parallelize([ x for x in range(10) ]).map(lambda x: x +
b.value.get()).collect()
File
"/Users/ed/.spark/spark-1.5.2-bin-hadoop2.6/python/lib/pyspark.zip/pyspark/broadcast.py",
line 97, in value
self._value = self.load(self._path)
File
"/Users/ed/.spark/spark-1.5.2-bin-hadoop2.6/python/lib/pyspark.zip/pyspark/broadcast.py",
line 88, in load
return pickle.load(f)
AttributeError: 'module' object has no attribute 'SimpleClass'
[snip]
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/PySpark-Broadcast-of-User-Defined-Class-No-Work-tp26000.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
