Hi, not sure about different environment but it is something that you can look into as well. Why would you need different environments for your job?
Regards, Gourav On Wed, Apr 17, 2019 at 11:43 AM Gorka Bravo Martinez < gorka.bravo.marti...@cern.ch> wrote: > Hi Gourav, > > you mean by seting a different python environment while running pyspark? > > Cheers, Gorka. > ________________________________________ > From: Gourav Sengupta [gourav.sengu...@gmail.com] > Sent: 17 April 2019 10:06 > To: Gorka Bravo Martinez > Cc: user@spark.apache.org > Subject: Re: Boto3 library send to pyspark > > Hi, > > there is addPyFile, and then there is python environment, try to search > for using python package managers like canopy and conda. > > Regards, > Gourav > > On Wed, Apr 17, 2019 at 8:50 AM Gorka Bravo Martinez < > gorka.bravo.marti...@cern.ch<mailto:gorka.bravo.marti...@cern.ch>> wrote: > Hi all, > > I would like to send a boto/boto3 library while running pyspark with yarn > client mode, how is it possible? > I am aware sc.addFile(<path>) can add a .py file, is it the same for a > library? > > Cheers, Gorka. >
