I am sorry for saying just based on my wild guess because I have no way to check and take a look into Jenkins but I think we might have to set the explicit Python version in https://github.com/apache/spark/blob/master/dev/run-tests-jenkins#L29
I guess we set the explicit Python version for running unit tests but it looks the script itself is dependent python executable in the path. On 29 Jul 2017 7:39 am, "Dong Joon Hyun" <dh...@hortonworks.com> wrote: I saw that error in the latest branch-2.1 build failure, too. https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Tes t%20(Dashboard)/job/spark-branch-2.1-test-sbt-hadoop-2.7/579/console But, the code was written in Jan 2016. Didn’t we run it on Python 2.6 without any problem? ee74498de37 (Josh Rosen 2016-01-26 14:20:11 -0800 124) {m: set(m.dependencies).intersection(modules_to_test) for m in modules_to_test}, sort=True) Bests, Dongjoon. *From: *Hyukjin Kwon <gurwls...@gmail.com> *Date: *Friday, July 28, 2017 at 7:06 AM *To: *Sean Owen <so...@cloudera.com> *Cc: *dev <dev@spark.apache.org> *Subject: *Re: Tests failing with run-tests.py SyntaxError Yes, that's my guess just given information here without a close look. On 28 Jul 2017 11:03 pm, "Sean Owen" <so...@cloudera.com> wrote: I see, does that suggest that a machine has 2.6, when it should use 2.7? On Fri, Jul 28, 2017 at 2:58 PM Hyukjin Kwon <gurwls...@gmail.com> wrote: That looks appearently due to dict comprehension which is, IIRC, not allowed in Python 2.6.x. I checked the release note for sure before - https://issues.apache.org/jira/browse/SPARK-20149 On 28 Jul 2017 9:56 pm, "Sean Owen" <so...@cloudera.com> wrote: File "./dev/run-tests.py", line 124 {m: set(m.dependencies).intersection(modules_to_test) for m in modules_to_test}, sort=True) ^ SyntaxError: invalid syntax It seems like tests are failing intermittently with this type of error, which leads me to guess there's some difference in the Python interpreter on one or more machines but not all (?) Does the error suggest anything to anyone who knows Python better than I? The line has been around for a year so I don't think it's the script per se.
