Yep, we're going to merge a change to separate the k8s tests into a separate profile, and fix up the Scala 2.12 thing. While non-critical those are pretty nice to have for 2.4. I think that's doable within the next 12 hours even.
@skonto I think there's one last minor thing needed on this PR? https://github.com/apache/spark/pull/22838/files#r228363727 On Thu, Oct 25, 2018 at 6:42 PM Wenchen Fan <cloud0...@gmail.com> wrote: > Any updates on this topic? https://github.com/apache/spark/pull/22827 is > merged and 2.4 is unblocked. > > I'll cut RC5 shortly after the weekend, and it will be great to include > the change proposed here. > > Thanks, > Wenchen > > On Fri, Oct 26, 2018 at 12:55 AM Stavros Kontopoulos < > stavros.kontopou...@lightbend.com> wrote: > >> I think it's worth getting in a change to just not enable this module, >>> which ought to be entirely safe, and avoid two of the issues we >>> identified. >>> >> >> Besides disabling it, when someone wants to run the tests with 2.12 he >> should be able to do so. So propagating the Scala profile still makes sense >> but it is not related to the release other than making sure things work >> fine. >> >> On Thu, Oct 25, 2018 at 7:02 PM, Sean Owen <sro...@gmail.com> wrote: >> >>> I think it's worth getting in a change to just not enable this module, >>> which ought to be entirely safe, and avoid two of the issues we >>> identified. >>> that said it didn't block RC4 so need not block RC5. >>> But should happen today if we're doing it. >>> On Thu, Oct 25, 2018 at 10:47 AM Xiao Li <gatorsm...@gmail.com> wrote: >>> > >>> > Hopefully, this will not delay RC5. Since this is not a blocker >>> ticket, RC5 will start if all the blocker tickets are resolved. >>> > >>> > Thanks, >>> > >>> > Xiao >>> > >>> > Sean Owen <sro...@gmail.com> 于2018年10月25日周四 上午8:44写道： >>> >> >>> >> Yes, I agree, and perhaps you are best placed to do that for 2.4.0 >>> RC5 :) >>> >> >>> >> On Thu, Oct 25, 2018 at 10:41 AM Stavros Kontopoulos >>> >> <stavros.kontopou...@lightbend.com> wrote: >>> >> > >>> >> > I agree these tests should be manual for now but should be run >>> somehow before a release to make sure things are working right? >>> >> > >>> >> > For the other issue: >>> https://issues.apache.org/jira/browse/SPARK-25835 . >>> >> > >>> >> > >>> >> > On Thu, Oct 25, 2018 at 6:29 PM, Stavros Kontopoulos < >>> stavros.kontopou...@lightbend.com> wrote: >>> >> >> >>> >> >> I will open a jira for the profile propagation issue and have a >>> look to fix it. >>> >> >> >>> >> >> Stavros >>> >> >> >>> >> >> On Thu, Oct 25, 2018 at 6:16 PM, Erik Erlandson < >>> eerla...@redhat.com> wrote: >>> >> >>> >>> >> >>> >>> >> >>> I would be comfortable making the integration testing manual for >>> now. A JIRA for ironing out how to make it reliable for automatic as a >>> goal for 3.0 seems like a good idea. >>> >> >>> >>> >> >>> On Thu, Oct 25, 2018 at 8:11 AM Sean Owen <sro...@gmail.com> >>> wrote: >>> >> >>>> >>> >> >>>> Forking this thread. >>> >> >>>> >>> >> >>>> Because we'll have another RC, we could possibly address these >>> two >>> >> >>>> issues. Only if we have a reliable change of course. >>> >> >>>> >>> >> >>>> Is it easy enough to propagate the -Pscala-2.12 profile? can't >>> hurt. >>> >> >>>> >>> >> >>>> And is it reasonable to essentially 'disable' >>> >> >>>> kubernetes/integration-tests by removing it from the kubernetes >>> >> >>>> profile? it doesn't mean it goes away, just means it's run >>> manually, >>> >> >>>> not automatically. Is that actually how it's meant to be used >>> anyway? >>> >> >>>> in the short term? given the discussion around its requirements >>> and >>> >> >>>> minikube and all that? >>> >> >>>> >>> >> >>>> (Actually, this would also 'solve' the Scala 2.12 build problem >>> too) >>> >> >>>> >>> >> >>>> On Tue, Oct 23, 2018 at 2:45 PM Sean Owen <sro...@gmail.com> >>> wrote: >>> >> >>>> > >>> >> >>>> > To be clear I'm currently +1 on this release, with much >>> commentary. >>> >> >>>> > >>> >> >>>> > OK, the explanation for kubernetes tests makes sense. Yes I >>> think we need to propagate the scala-2.12 build profile to make it work. Go >>> for it, if you have a lead on what the change is. >>> >> >>>> > This doesn't block the release as it's an issue for tests, and >>> only affects 2.12. However if we had a clean fix for this and there were >>> another RC, I'd include it. >>> >> >>>> > >>> >> >>>> > Dongjoon has a good point about the >>> spark-kubernetes-integration-tests artifact. That doesn't sound like it >>> should be published in this way, though, of course, we publish the test >>> artifacts from every module already. This is only a bit odd in being a >>> non-test artifact meant for testing. But it's special testing! So I also >>> don't think that needs to block a release. >>> >> >>>> > >>> >> >>>> > This happens because the integration tests module is enabled >>> with the 'kubernetes' profile too, and also this output is copied into the >>> release tarball at kubernetes/integration-tests/tests. Do we need that in a >>> binary release? >>> >> >>>> > >>> >> >>>> > If these integration tests are meant to be run ad hoc, >>> manually, not part of a normal test cycle, then I think we can just not >>> enable it with -Pkubernetes. If it is meant to run every time, then it >>> sounds like we need a little extra work shown in recent PRs to make that >>> easier, but then, this test code should just be the 'test' artifact parts >>> of the kubernetes module, no? >>> >> >>>> >>> >> >>>> >>> --------------------------------------------------------------------- >>> >> >>>> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org >>> >> >>>> >>> >> >> >>> >> >> >>> >> > >>> >> >>> >> --------------------------------------------------------------------- >>> >> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org >>> >> >>> >> >> >> >> -- >> Stavros Kontopoulos >> >> *Senior Software Engineer* >> *Lightbend, Inc.* >> >> *p: +30 6977967274 <%2B1%20650%20678%200020>* >> *e: stavros.kontopou...@lightbend.com* <dave.mar...@lightbend.com> >> >> >>
