i'm ready to update the ubuntu workers/minikube/k8s to support the 4.1.2 client: https://issues.apache.org/jira/browse/SPARK-2674
i am more than comfortable with this build system update, both on the ops and spark project side. we were incredibly far behind the release cycle for k8s and minikube, which was beginning to impact the dep graph. updating to at least k8s v1.13 and the 4.1.2 client lib gives us a lot of breathing room w/little worry about backwards compatibility. if this is something we're comfortable with doing for the 2.4.1 release (+master), then i'll need to take down the pull request builder for ~30 mins (which will be it's own email to dev@). shane On Wed, Mar 6, 2019 at 12:40 PM Stavros Kontopoulos < stavros.kontopou...@lightbend.com> wrote: > Yes its a touch decision and as we discussed today ( > https://docs.google.com/document/d/1pnF38NF6N5eM8DlK088XUW85Vms4V2uTsGZvSp8MNIA > ) > "Kubernetes support window is 9 months, Spark is two years". So we may > end up with old client versions on branches still supported like 2.4.x in > the future. > That gives us no choice but to upgrade, if we want to be on the safe side. > We have tested 3.0.0 with 1.11 internally and it works but I dont know what > it means to run with old > clients. > > > On Wed, Mar 6, 2019 at 7:54 PM Sean Owen <sro...@gmail.com> wrote: > >> If the old client is basically unusable with the versions of K8S >> people mostly use now, and the new client still works with older >> versions, I could see including this in 2.4.1. >> >> Looking at >> https://github.com/fabric8io/kubernetes-client#compatibility-matrix >> it seems like the 4.1.1 client is needed for 1.10 and above. However >> it no longer supports 1.7 and below. >> We have 3.0.x, and versions through 4.0.x of the client support the >> same K8S versions, so no real middle ground here. >> >> 1.7.0 came out June 2017, it seems. 1.10 was March 2018. Minor release >> branches are maintained for 9 months per >> https://kubernetes.io/docs/setup/version-skew-policy/ >> >> Spark 2.4.0 came in Nov 2018. I suppose we could say it should have >> used the newer client from the start as at that point (?) 1.7 and >> earlier were already at least 7 months past EOL. >> If we update the client in 2.4.1, versions of K8S as recently >> 'supported' as a year ago won't work anymore. I'm guessing there are >> still 1.7 users out there? That wasn't that long ago but if the >> project and users generally move fast, maybe not. >> >> Normally I'd say, that's what the next minor release of Spark is for; >> update if you want later infra. But there is no Spark 2.5. >> I presume downstream distros could modify the dependency easily (?) if >> needed and maybe already do. It wouldn't necessarily help end users. >> >> Does the 3.0.x client not work at all with 1.10+ or just unsupported. >> If it 'basically works but no guarantees' I'd favor not updating. If >> it doesn't work at all, hm. That's tough. I think I'd favor updating >> the client but think it's a tough call both ways. >> >> >> >> On Wed, Mar 6, 2019 at 11:14 AM Stavros Kontopoulos >> <stavros.kontopou...@lightbend.com> wrote: >> > >> > Yes Shane Knapp has done the work for that already, and also tests >> pass, I am working on a PR now, I could submit it for the 2.4 branch . >> > I understand that this is a major dependency update, but the problem I >> see is that the client version is so old that I dont think it makes >> > much sense for current users who are on k8s 1.10, 1.11 etc( >> https://github.com/fabric8io/kubernetes-client#compatibility-matrix, >> 3.0.0 does not even exist in there). >> > I dont know what it means to use that old version with current k8s >> clusters in terms of bugs etc. >> > > > -- Shane Knapp UC Berkeley EECS Research / RISELab Staff Technical Lead https://rise.cs.berkeley.edu
