Re: [discuss] 2.4.1-rcX release, k8s client PRs, build system infrastructure update

2019-03-14 Thread shane knapp
about to send mail. On Thu, Mar 14, 2019 at 1:00 PM Marcelo Vanzin wrote: > Not sure if anyone else is having the same issues, but jenkins seems > to be in a weird state. > > I can't connect to the web UI, and it doesn't seem to be responding to > test requests. > > On Wed, Mar 13, 2019 at 3:15

Re: [discuss] 2.4.1-rcX release, k8s client PRs, build system infrastructure update

2019-03-14 Thread Marcelo Vanzin
Not sure if anyone else is having the same issues, but jenkins seems to be in a weird state. I can't connect to the web UI, and it doesn't seem to be responding to test requests. On Wed, Mar 13, 2019 at 3:15 PM shane knapp wrote: > > upgrade completed, jenkins building again... master PR merged

Re: [discuss] 2.4.1-rcX release, k8s client PRs, build system infrastructure update

2019-03-14 Thread shane knapp
i think we should. the version of k8s we were testing against before was v1.10.x, and we are currently testing master *and* 2.4.1 against v1.13.1. given that we're a couple of weeks away from v.1.14.x i feel that this is a safe and prudent thing to include. On Thu, Mar 14, 2019 at 10:57 AM DB T

Re: [discuss] 2.4.1-rcX release, k8s client PRs, build system infrastructure update

2019-03-14 Thread Marcelo Vanzin
IMO we can vote on rc8; that upgrade should not block it. If it fails for some other reason, then rc9 will have it. Supported versions in k8s-land are really weird and don't match well to our release schedule. There isn't a good solution to this at the moment... good thing k8s support is still mar

Re: [discuss] 2.4.1-rcX release, k8s client PRs, build system infrastructure update

2019-03-14 Thread DB Tsai
Since rc8 was already cut without the k8s client upgrade; the build is ready to vote, and including k8s client upgrade in 2.4.1 implies that we will drop the old-but-not-that-old K8S versions as Sean mentioned, should we include this upgrade in 2.4.2? Thanks. Sincerely, DB Tsai -

Re: [discuss] 2.4.1-rcX release, k8s client PRs, build system infrastructure update

2019-03-14 Thread shane knapp
thanks everyone, both PRs are merged. :) On Wed, Mar 13, 2019 at 3:51 PM shane knapp wrote: > btw, let's wait and see if the non-k8s PRB tests pass before merging > https://github.com/apache/spark/pull/23993 in to 2.4.1 > > On Wed, Mar 13, 2019 at 3:42 PM shane knapp wrote: > >> 2.4.1 k8s inte

Re: [discuss] 2.4.1-rcX release, k8s client PRs, build system infrastructure update

2019-03-13 Thread shane knapp
btw, let's wait and see if the non-k8s PRB tests pass before merging https://github.com/apache/spark/pull/23993 in to 2.4.1 On Wed, Mar 13, 2019 at 3:42 PM shane knapp wrote: > 2.4.1 k8s integration test passed: > > > https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribu

Re: [discuss] 2.4.1-rcX release, k8s client PRs, build system infrastructure update

2019-03-13 Thread shane knapp
2.4.1 k8s integration test passed: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/8875/ thanks everyone! :) On Wed, Mar 13, 2019 at 3:24 PM shane knapp wrote: > 2.4.1 integration tests running: > https://amplab.cs.berkeley.edu/jenkins/job/testing-k8

Re: [discuss] 2.4.1-rcX release, k8s client PRs, build system infrastructure update

2019-03-13 Thread shane knapp
2.4.1 integration tests running: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/8875/ On Wed, Mar 13, 2019 at 3:15 PM shane knapp wrote: > upgrade completed, jenkins building again... master PR merged, waiting > for the 2.4.1 PR to launch the k8s inte

Re: [discuss] 2.4.1-rcX release, k8s client PRs, build system infrastructure update

2019-03-13 Thread shane knapp
upgrade completed, jenkins building again... master PR merged, waiting for the 2.4.1 PR to launch the k8s integration tests. On Wed, Mar 13, 2019 at 2:55 PM shane knapp wrote: > okie dokie! the time approacheth! > > i'll pause jenkins @ 3pm to not accept new jobs. i don't expect the > upgrade

Re: [discuss] 2.4.1-rcX release, k8s client PRs, build system infrastructure update

2019-03-13 Thread shane knapp
okie dokie! the time approacheth! i'll pause jenkins @ 3pm to not accept new jobs. i don't expect the upgrade to take more than 15-20 mins, following which i will re-enable builds. On Wed, Mar 13, 2019 at 12:17 PM shane knapp wrote: > ok awesome. let's shoot for 3pm PST. > > On Wed, Mar 13,

Re: [discuss] 2.4.1-rcX release, k8s client PRs, build system infrastructure update

2019-03-13 Thread Marcelo Vanzin
Sounds good. On Wed, Mar 13, 2019 at 12:17 PM shane knapp wrote: > > ok awesome. let's shoot for 3pm PST. > > On Wed, Mar 13, 2019 at 11:59 AM Marcelo Vanzin wrote: >> >> On Wed, Mar 13, 2019 at 11:53 AM shane knapp wrote: >> > On Wed, Mar 13, 2019 at 11:49 AM Marcelo Vanzin >> > wrote: >> >

Re: [discuss] 2.4.1-rcX release, k8s client PRs, build system infrastructure update

2019-03-13 Thread shane knapp
ok awesome. let's shoot for 3pm PST. On Wed, Mar 13, 2019 at 11:59 AM Marcelo Vanzin wrote: > On Wed, Mar 13, 2019 at 11:53 AM shane knapp wrote: > > On Wed, Mar 13, 2019 at 11:49 AM Marcelo Vanzin > wrote: > >> > >> Do the upgraded minikube/k8s versions break the current master client > >> v

Re: [discuss] 2.4.1-rcX release, k8s client PRs, build system infrastructure update

2019-03-13 Thread Sean Owen
I'm OK with this take. The problem with back-porting the client update to 2.4.x at all is that it drops support for some old-but-not-that-old K8S versions, which feels surprising in a maintenance release. That said, maybe it's OK, and a little more OK for a 2.4.2 in several months' time. On Wed, M

Re: [discuss] 2.4.1-rcX release, k8s client PRs, build system infrastructure update

2019-03-13 Thread Marcelo Vanzin
On Wed, Mar 13, 2019 at 11:53 AM shane knapp wrote: > On Wed, Mar 13, 2019 at 11:49 AM Marcelo Vanzin wrote: >> >> Do the upgraded minikube/k8s versions break the current master client >> version too? >> > yes. Ah, so that part kinda sucks. Let's do this: since the master PR is good to go pendi

Re: [discuss] 2.4.1-rcX release, k8s client PRs, build system infrastructure update

2019-03-13 Thread shane knapp
On Wed, Mar 13, 2019 at 11:49 AM Marcelo Vanzin wrote: > Do the upgraded minikube/k8s versions break the current master client > version too? > > yes. > I'm not super concerned about 2.4 integration tests being broken for a > little bit. It's very uncommon for new PRs to be open against > branc

Re: [discuss] 2.4.1-rcX release, k8s client PRs, build system infrastructure update

2019-03-13 Thread Marcelo Vanzin
Do the upgraded minikube/k8s versions break the current master client version too? I'm not super concerned about 2.4 integration tests being broken for a little bit. It's very uncommon for new PRs to be open against branch-2.4 that would affect k8s. But I really don't want master to break. So if

[discuss] 2.4.1-rcX release, k8s client PRs, build system infrastructure update

2019-03-13 Thread shane knapp
hey everyone... i wanted to break this discussion out of the mega-threads for the 2.4.1 RC candidates. the TL;DR is that we've been trying to update the k8s client libs to something much more modern. however, for us to do this, we need to update our very old k8s and minikube versions. the probl