initially, i'd like to just choose one version to have the primary tests against, but i'm also not opposed to supporting more of a matrix. the biggest problem i see w/this approach, however, is that of build monitoring and long-term ownership. this is why we have a relatively restrictive current deployment.
another thing i've been noticing during this project is that we have a lot of flaky tests... for instance, i'm literally having every other build fail on my (relatively) up-to-date PRB fork: https://amplab.cs.berkeley.edu/jenkins/job/ubuntuSparkPRB/ (i'm testing more than python here, otherwise i could just build a spark distro and run the python tests against that) i'll also set up a 3.6 env tomorrow and start testing against that. i'm pretty confident about 3.5, tho. shane On Mon, Aug 20, 2018 at 11:33 AM, Bryan Cutler <cutl...@gmail.com> wrote: > Thanks for looking into this Shane! If we are choosing a single python > 3.x, I think 3.6 would be good. It might still be nice to test against > other versions too, so we can catch any issues. Is it possible to have more > exhaustive testing as part of a nightly or scheduled build? As a point of > reference for Python 3.6, Arrow is using this version for CI. > > Bryan > > On Sun, Aug 19, 2018 at 9:49 PM Hyukjin Kwon <gurwls...@gmail.com> wrote: > >> Actually Python 3.7 is released (https://www.python.org/ >> downloads/release/python-370/) too and I fixed the compatibility issues >> accordingly - https://github.com/apache/spark/pull/21714 >> There has been an issue for 3.6 (comparing to lower versions of Python >> including 3.5) - https://github.com/apache/spark/pull/16429 >> >> I am not yet sure what's the best matrix for it actually. In case of R, >> we test lowest version in Jenkins and highest version via AppVeyor FWIW. >> I don't have a strong preference opinion on this since we have been >> having compatibility issues for each Python version. >> >> >> 2018년 8월 14일 (화) 오전 4:15, shane knapp <skn...@berkeley.edu>님이 작성: >> >>> hey everyone! >>> >>> i was checking out the EOL/release cycle for python 3.5 and it looks >>> like we'll have 3.5.6 released in early 2019. >>> >>> this got me to thinking: instead of 3.5, what about 3.6? >>> >>> i looked around, and according to the 'docs' and 'collective wisdom of >>> the internets', 3.5 and 3.6 should be fully backwards-compatible w/3.4. >>> >>> of course, this needs to be taken w/a grain of salt, as we're mostly >>> focused on actual python package requirements, rather than worrying about >>> core python functionality. >>> >>> thoughts? comments? >>> >>> thanks in advance, >>> >>> shane >>> -- >>> Shane Knapp >>> UC Berkeley EECS Research / RISELab Staff Technical Lead >>> https://rise.cs.berkeley.edu >>> >> -- Shane Knapp UC Berkeley EECS Research / RISELab Staff Technical Lead https://rise.cs.berkeley.edu
