I see. I will try the other way around. On Thu, Mar 19, 2015 at 8:06 PM, Davies Liu <[email protected]> wrote:
> the options of spark-submit should come before main.py, or they will > become the options of main.py, so it should be: > > ../hadoop/spark-install/bin/spark-submit --py-files > > > /home/poiuytrez/naive.py,/home/poiuytrez/processing.py,/home/poiuytrez/settings.py > --master spark://spark-m:7077 main.py > > On Mon, Mar 16, 2015 at 4:11 AM, poiuytrez <[email protected]> > wrote: > > I have a spark app which is composed of multiple files. > > > > When I launch Spark using: > > > > ../hadoop/spark-install/bin/spark-submit main.py --py-files > > > /home/poiuytrez/naive.py,/home/poiuytrez/processing.py,/home/poiuytrez/settings.py > > --master spark://spark-m:7077 > > > > I am getting an error: > > > > 15/03/13 15:54:24 INFO TaskSetManager: Lost task 6.3 in stage 413.0 > (TID > > 5817) on executor spark-w-3.c.databerries.internal: > > org.apache.spark.api.python.PythonException (Traceback (most recent call > > last): File "/home/hadoop/spark-install/python/pyspark/worker.py", line > > 90, in main > > command = pickleSer._read_with_length(infile) File > > "/home/hadoop/spark-install/python/pyspark/serializers.py", line 151, in > > _read_with_length > > return self.loads(obj) File > > "/home/hadoop/spark-install/python/pyspark/serializers.py", line 396, in > > loads > > return cPickle.loads(obj) ImportError: No module named naive > > > > It is weird because I do not serialize anything. naive.py is also > available > > on every machine at the same path. > > > > Any insight on what could be going on? The issue does not happen on my > > laptop. > > > > PS : I am using Spark 1.2.0. > > > > > > > > -- > > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/Error-when-using-multiple-python-files-spark-submit-tp22080.html > > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: [email protected] > > For additional commands, e-mail: [email protected] > > >
