Hi I am very new both in spark and aws stuff.. Say, I want to install pandas on ec2.. (pip install pandas) How do I create the image and the above library which would be used from pyspark. Thanks
On Sun, Feb 8, 2015 at 3:03 AM, gen tang <gen.tan...@gmail.com> wrote: > Hi, > > You can make a image of ec2 with all the python libraries installed and > create a bash script to export python_path in the /etc/init.d/ directory. > Then you can launch the cluster with this image and ec2.py > > Hope this can be helpful > > Cheers > Gen > > > On Sun, Feb 8, 2015 at 9:46 AM, Chengi Liu <chengi.liu...@gmail.com> > wrote: > >> Hi, >> I want to install couple of python libraries (pip install >> python_library) which I want to use on pyspark cluster which are developed >> using the ec2 scripts. >> Is there a way to specify these libraries when I am building those ec2 >> clusters? >> Whats the best way to install these libraries on each ec2 node? >> Thanks >> > >
