When you run spark submit in either client or cluster mode, you can either use
the options --packages or -jars to automatically copy your packages to the
worker machines.
Thanks
On Monday, January 11, 2016 12:52 PM, Andy Davidson
<[email protected]> wrote:
I use https://code.google.com/p/parallel-ssh/ to upgrade all my slaves
From: "taotao.li" <[email protected]>
Date: Sunday, January 10, 2016 at 9:50 PM
To: "user @spark" <[email protected]>
Subject: pre-install 3-party Python package on spark cluster
I have a spark cluster, from machine-1 to machine 100, and machine-1 acts asthe
master.
Then one day my program need use a 3-party python package which is notinstalled
on every machine of the cluster.
so here comes my problem: to make that 3-party python package usable onmaster
and slaves, should I manually ssh to every machine and use pip toinstall that
package?
I believe there should be some deploy scripts or other things to make
thisgrace, but I can't find anything after googling.
--View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/pre-install-3-party-Python-package-on-spark-cluster-tp25930.htmlSent
from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------To
unsubscribe, e-mail: [email protected] additional commands,
e-mail: [email protected]
