The function run in worker is serialized in driver, so the driver and worker should be run in the same Python interpreter.
If you do not need c extension support, then Jython will be better than CPython, because of the cost of serialization is much lower. Davies -- View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/PySpark-Driver-from-Jython-tp7142p7269.html Sent from the Apache Spark Developers List mailing list archive at Nabble.com.
