https://issues.apache.org/jira/browse/SPARK-26742
On Thu, Mar 7, 2019 at 10:52 AM shane knapp <skn...@berkeley.edu> wrote: > i'm ready to update the ubuntu workers/minikube/k8s to support the 4.1.2 > client: > https://issues.apache.org/jira/browse/SPARK-2674 > > i am more than comfortable with this build system update, both on the ops > and spark project side. we were incredibly far behind the release cycle > for k8s and minikube, which was beginning to impact the dep graph. > updating to at least k8s v1.13 and the 4.1.2 client lib gives us a lot of > breathing room w/little worry about backwards compatibility. > > if this is something we're comfortable with doing for the 2.4.1 release > (+master), then i'll need to take down the pull request builder for ~30 > mins (which will be it's own email to dev@). > > shane > > On Wed, Mar 6, 2019 at 12:40 PM Stavros Kontopoulos < > stavros.kontopou...@lightbend.com> wrote: > >> Yes its a touch decision and as we discussed today ( >> https://docs.google.com/document/d/1pnF38NF6N5eM8DlK088XUW85Vms4V2uTsGZvSp8MNIA >> ) >> "Kubernetes support window is 9 months, Spark is two years". So we may >> end up with old client versions on branches still supported like 2.4.x in >> the future. >> That gives us no choice but to upgrade, if we want to be on the safe >> side. We have tested 3.0.0 with 1.11 internally and it works but I dont >> know what it means to run with old >> clients. >> >> >> On Wed, Mar 6, 2019 at 7:54 PM Sean Owen <sro...@gmail.com> wrote: >> >>> If the old client is basically unusable with the versions of K8S >>> people mostly use now, and the new client still works with older >>> versions, I could see including this in 2.4.1. >>> >>> Looking at >>> https://github.com/fabric8io/kubernetes-client#compatibility-matrix >>> it seems like the 4.1.1 client is needed for 1.10 and above. However >>> it no longer supports 1.7 and below. >>> We have 3.0.x, and versions through 4.0.x of the client support the >>> same K8S versions, so no real middle ground here. >>> >>> 1.7.0 came out June 2017, it seems. 1.10 was March 2018. Minor release >>> branches are maintained for 9 months per >>> https://kubernetes.io/docs/setup/version-skew-policy/ >>> >>> Spark 2.4.0 came in Nov 2018. I suppose we could say it should have >>> used the newer client from the start as at that point (?) 1.7 and >>> earlier were already at least 7 months past EOL. >>> If we update the client in 2.4.1, versions of K8S as recently >>> 'supported' as a year ago won't work anymore. I'm guessing there are >>> still 1.7 users out there? That wasn't that long ago but if the >>> project and users generally move fast, maybe not. >>> >>> Normally I'd say, that's what the next minor release of Spark is for; >>> update if you want later infra. But there is no Spark 2.5. >>> I presume downstream distros could modify the dependency easily (?) if >>> needed and maybe already do. It wouldn't necessarily help end users. >>> >>> Does the 3.0.x client not work at all with 1.10+ or just unsupported. >>> If it 'basically works but no guarantees' I'd favor not updating. If >>> it doesn't work at all, hm. That's tough. I think I'd favor updating >>> the client but think it's a tough call both ways. >>> >>> >>> >>> On Wed, Mar 6, 2019 at 11:14 AM Stavros Kontopoulos >>> <stavros.kontopou...@lightbend.com> wrote: >>> > >>> > Yes Shane Knapp has done the work for that already, and also tests >>> pass, I am working on a PR now, I could submit it for the 2.4 branch . >>> > I understand that this is a major dependency update, but the problem I >>> see is that the client version is so old that I dont think it makes >>> > much sense for current users who are on k8s 1.10, 1.11 etc( >>> https://github.com/fabric8io/kubernetes-client#compatibility-matrix, >>> 3.0.0 does not even exist in there). >>> > I dont know what it means to use that old version with current k8s >>> clusters in terms of bugs etc. >>> >> >> >> > > -- > Shane Knapp > UC Berkeley EECS Research / RISELab Staff Technical Lead > https://rise.cs.berkeley.edu > -- Shane Knapp UC Berkeley EECS Research / RISELab Staff Technical Lead https://rise.cs.berkeley.edu
