ok, I see now what's happening - the pkg.mod.test is serialized by reference
and there is nothing actually trying to import pkg.mod on the executors so the
reference is broken.
so how can I get the pkg.mod imported on the executors?
thanks,Antony.
On Friday, 2 January 2015, 13:49, Antony M
Hi,
I am running spark 1.1.0 on yarn. I have custom set of modules installed under
same location on each executor node and wondering how can I pass the executors
the PYTHONPATH so that they can use the modules.
I've tried this:
spark-env.sh:export PYTHONPATH=/tmp/test/
spark-defaults.conf:spark.
