after spending the last three hours unraveling how dev/create-release/release-build.sh works, i think i figured out what needs to happen.
1) https://issues.apache.org/jira/browse/SPARK-26565 2) create world-wide campaign to stop bash scripting my brain... it hurts. On Sun, Jan 6, 2019 at 1:07 PM Dongjoon Hyun <dongjoon.h...@gmail.com> wrote: > Thank you, Felix and Shane. > > +1 for packaging except signing in Jenkins. That sounds perfect for the > next releases (2.2.3, 2.3.3, 3.0.0). > > BTW, thanks to Shane, the followings are recovered from today. > > 1. For all live branches (master to branch-2.2), Jenkins Packaging jobs > are recovered to build and packaging. Only there exist signing errors. > > - Master: > https://amplab.cs.berkeley.edu/jenkins/view/Spark%20Packaging/job/spark-master-package/2181/console > - branch-2.4: > https://amplab.cs.berkeley.edu/jenkins/view/Spark%20Packaging/job/spark-branch-2.4-package/95/console > - branch-2.3: > https://amplab.cs.berkeley.edu/jenkins/view/Spark%20Packaging/job/spark-branch-2.3-package/230/console > - branch-2.2: > https://amplab.cs.berkeley.edu/jenkins/view/Spark%20Packaging/job/spark-branch-2.2-package/240/console > > 2. Daily snapshot maven publishing is recovered. This is helpful for > partial module testing. > > - > https://amplab.cs.berkeley.edu/jenkins/view/Spark%20Packaging/job/spark-master-maven-snapshots/ > > Thank you all. Everything becomes healthier during 2.2.3 release. > This will reduce the burden of 2.3.3/3.0.0 releases. :D > > Bests, > Dongjoon. > > > On Sun, Jan 6, 2019 at 11:42 AM Felix Cheung <felixcheun...@hotmail.com> > wrote: > >> Awesome Shane! >> >> >> ------------------------------ >> *From:* shane knapp <skn...@berkeley.edu> >> *Sent:* Sunday, January 6, 2019 11:38 AM >> *To:* Felix Cheung >> *Cc:* Dongjoon Hyun; Wenchen Fan; dev >> *Subject:* Re: Spark Packaging Jenkins >> >> noted. i like the idea of building (but not signing) the release and >> will update the job(s) this week. >> >> On Sun, Jan 6, 2019 at 11:22 AM Felix Cheung <felixcheun...@hotmail.com> >> wrote: >> >>> https://spark.apache.org/release-process.html >>> >>> Look for do-release-docker.sh script >>> >>> >>> ------------------------------ >>> *From:* Felix Cheung <felixcheun...@hotmail.com> >>> *Sent:* Sunday, January 6, 2019 11:17 AM >>> *To:* Dongjoon Hyun; Wenchen Fan >>> *Cc:* dev; shane knapp >>> *Subject:* Re: Spark Packaging Jenkins >>> >>> The release process doc should have been updated on this - as mentioned >>> we do not use Jenkins for release signing (take this offline if further >>> discussion is needed) >>> >>> The release build on Jenkins can still be useful for pre-validating the >>> release build process (without actually signing it) >>> >>> >>> ------------------------------ >>> *From:* Dongjoon Hyun <dongjoon.h...@gmail.com> >>> *Sent:* Saturday, January 5, 2019 9:46 PM >>> *To:* Wenchen Fan >>> *Cc:* dev; shane knapp >>> *Subject:* Re: Spark Packaging Jenkins >>> >>> Thank you, Wenchen. >>> >>> I see. I'll update the doc and proceed to the next step manually as you >>> advise. And it seems that we can stop the outdated Jenkins jobs, too. >>> >>> Bests, >>> Dongjoon. >>> >>> On Sat, Jan 5, 2019 at 20:15 Wenchen Fan <cloud0...@gmail.com> wrote: >>> >>>> IIRC there was a change to the release process: we stop using the >>>> shared gpg key on Jenkins, but use the personal key of the release manager. >>>> I'm not sure Jenkins can help testing package anymore. >>>> >>>> BTW release manager needs to run the packaging script by himself. If >>>> there is a problem, the release manager will find it out sooner or later. >>>> >>>> >>>> >>>> On Sun, Jan 6, 2019 at 6:34 AM Dongjoon Hyun <dongjoon.h...@gmail.com> >>>> wrote: >>>> >>>>> Hi, All. >>>>> >>>>> It turns out that `gpg signing` is the next huddle in Spark Packaging >>>>> Jenkins. >>>>> Since 2.4.0 release, is there something changed in our Jenkins machine? >>>>> >>>>> gpg: skipped >>>>> "/home/jenkins/workspace/spark-master-package/spark-utils/new-release-scripts/jenkins/jenkins-credentials-JEtz0nyn/gpg.tmp": >>>>> No secret key >>>>> gpg: signing failed: No secret key >>>>> >>>>> Bests, >>>>> Dongjoon. >>>>> >>>>> >>>>> On Fri, Jan 4, 2019 at 11:52 AM shane knapp <skn...@berkeley.edu> >>>>> wrote: >>>>> >>>>>> https://issues.apache.org/jira/browse/SPARK-26537 >>>>>> >>>>>> On Fri, Jan 4, 2019 at 11:31 AM shane knapp <skn...@berkeley.edu> >>>>>> wrote: >>>>>> >>>>>>> this may push in to early next week... these builds were set up >>>>>>> before my time, and i'm currently unraveling how they all work before >>>>>>> pushing a commit to fix stuff. >>>>>>> >>>>>>> nothing like some code archaeology to make my friday more exciting! >>>>>>> :) >>>>>>> >>>>>>> shane >>>>>>> >>>>>>> On Fri, Jan 4, 2019 at 11:08 AM Dongjoon Hyun < >>>>>>> dongjoon.h...@gmail.com> wrote: >>>>>>> >>>>>>>> Thank you, Shane! >>>>>>>> >>>>>>>> Bests, >>>>>>>> Dongjoon. >>>>>>>> >>>>>>>> On Fri, Jan 4, 2019 at 10:50 AM shane knapp <skn...@berkeley.edu> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> yeah, i'll get on that today. thanks for the heads up. >>>>>>>>> >>>>>>>>> On Fri, Jan 4, 2019 at 10:46 AM Dongjoon Hyun < >>>>>>>>> dongjoon.h...@gmail.com> wrote: >>>>>>>>> >>>>>>>>>> Hi, All >>>>>>>>>> >>>>>>>>>> As a part of release process, we need to check >>>>>>>>>> Packaging/Compile/Test Jenkins status. >>>>>>>>>> >>>>>>>>>> http://spark.apache.org/release-process.html >>>>>>>>>> >>>>>>>>>> 1. Spark Packaging: >>>>>>>>>> https://amplab.cs.berkeley.edu/jenkins/view/Spark%20Packaging/ >>>>>>>>>> 2. Spark QA Compile: >>>>>>>>>> https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Compile/ >>>>>>>>>> 3. Spark QA Test: >>>>>>>>>> https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test/ >>>>>>>>>> >>>>>>>>>> Currently, (2) and (3) are working because it uses GitHub ( >>>>>>>>>> https://github.com/apache/spark.git). >>>>>>>>>> But, (1) seems to be broken because it's looking for old repo( >>>>>>>>>> https://git-wip-us.apache.org/repos/asf/spark.git/info/refs) >>>>>>>>>> instead of new GitBox. >>>>>>>>>> >>>>>>>>>> Can we fix this in this week? >>>>>>>>>> >>>>>>>>>> Bests, >>>>>>>>>> Dongjoon. >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Shane Knapp >>>>>>>>> UC Berkeley EECS Research / RISELab Staff Technical Lead >>>>>>>>> https://rise.cs.berkeley.edu >>>>>>>>> >>>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Shane Knapp >>>>>>> UC Berkeley EECS Research / RISELab Staff Technical Lead >>>>>>> https://rise.cs.berkeley.edu >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Shane Knapp >>>>>> UC Berkeley EECS Research / RISELab Staff Technical Lead >>>>>> https://rise.cs.berkeley.edu >>>>>> >>>>> >> >> -- >> Shane Knapp >> UC Berkeley EECS Research / RISELab Staff Technical Lead >> https://rise.cs.berkeley.edu >> > -- Shane Knapp UC Berkeley EECS Research / RISELab Staff Technical Lead https://rise.cs.berkeley.edu
