thanks for the heads up... i'll test deploy this tomorrow and see what gotchas turn up. we may need to upgrade from python 3.4 to 3.5 IIRC.
On Mon, Mar 25, 2019 at 6:16 PM Hyukjin Kwon <gurwls...@gmail.com> wrote: > Hi all, > > We really need to upgrade the minimal version soon. It's actually slowing > down the PySpark dev, for instance, by the overhead that sometimes we need > currently to test all multiple matrix of Arrow and Pandas. Also, it > currently requires to add some weird hacks or ugly codes. Some bugs exist > in lower versions, and some features are not supported in low PyArrow, for > instance. > > Per, (Apache Arrow'+ Spark committer FWIW), Bryan's recommendation and my > opinion as well, we should better increase the minimal version to 0.12.x. > (Also, note that Pandas <> Arrow is an experimental feature). > > So, I and Bryan will proceed this roughly in few days if there isn't > objections assuming we're fine with increasing it to 0.12.x. Please let me > know if there are some concerns. > > For clarification, this requires some jobs in Jenkins to upgrade the > minimal version of PyArrow (I cc'ed Shane as well). > > PS: I roughly heard that Shane's busy for some work stuff .. but it's kind > of important in my perspective. > > -- Shane Knapp UC Berkeley EECS Research / RISELab Staff Technical Lead https://rise.cs.berkeley.edu
