Perfect. Thanks Josh. I've added myself as a watcher on the ticket. (By the way, when I upgraded to 2.7 I replaced 2.6 so the executable name didn't change.)
On Fri, Feb 28, 2014 at 12:12 AM, Josh Rosen <rosenvi...@gmail.com> wrote: > There's an open ticket to update the Python version: > https://spark-project.atlassian.net/browse/SPARK-922. In that ticket, I > included instructions for a workaround to manually update a cluster to > Python 2.7. > > Did you set the PYSPARK_PYTHON environment variable to the name of your > new Python executable? You may have to do this if Python 2.7 executable > was installed as "python27" instead of "python." > > > On Thu, Feb 27, 2014 at 9:03 PM, Nicholas Chammas < > nicholas.cham...@gmail.com> wrote: > >> Makes sense. I'll give it a shot and check back here if that doesn't work. >> >> Are there plans to upgrade the EC2 deployment scripts and/or AMI to have >> Python 2.7 by default? If so, is there a ticket somewhere I can follow? >> >> Nick >> >> >> On Thu, Feb 27, 2014 at 6:50 PM, Bryn Keller <xol...@xoltar.org> wrote: >> >>> Hi Nick, >>> >>> All the nodes of the cluster need to have the same Python setup (path >>> and version). So if, e.g. you start running in 2.7.5 on the master and it >>> ships code to nodes that have 2.6.x, you'll get invalid opcode errors. >>> >>> Thanks, >>> Bryn >>> >>> >>> On Thu, Feb 27, 2014 at 3:48 PM, nicholas.chammas < >>> nicholas.cham...@gmail.com> wrote: >>> >>>> The provided Spark EC2 >>>> scripts<https://spark.incubator.apache.org/docs/0.9.0/ec2-scripts.html>and >>>> default AMI ship with Python 2.6.8. >>>> >>>> I would like to use Python 2.7.5 or later. I believe that among the 2.x >>>> versions, 2.7 is the most popular. >>>> >>>> What's the easiest way to get my Spark cluster on Python 2.7 and >>>> working correctly? >>>> >>>> For starters, I naively upgraded Python on the master node using the >>>> instructions >>>> here<http://www.lecloud.net/post/61401763496/install-update-to-python-2-7-and-latest-pip-on-ec2>. >>>> The pyspark shell started up fine but a basic map on an RDD that used to >>>> work fine failed. Perhaps I just need to upgrade all the nodes in my >>>> cluster, and perhaps it's not that simple. I thought I'd check here first. >>>> >>>> Nick >>>> >>>> >>>> ------------------------------ >>>> View this message in context: Running Spark with Python >>>> 2.7.5+<http://apache-spark-user-list.1001560.n3.nabble.com/Running-Spark-with-Python-2-7-5-tp2131.html> >>>> Sent from the Apache Spark User List mailing list >>>> archive<http://apache-spark-user-list.1001560.n3.nabble.com/>at Nabble.com. >>>> >>> >>> >> >
