Re: Spark Packaging Jenkins
Hi, All. It turns out that `gpg signing` is the next huddle in Spark Packaging Jenkins. Since 2.4.0 release, is there something changed in our Jenkins machine? gpg: skipped "/home/jenkins/workspace/spark-master-package/spark-utils/new-release-scripts/jenkins/jenkins-credentials-JEtz0nyn/gpg.tmp": No secret key gpg: signing failed: No secret key Bests, Dongjoon. On Fri, Jan 4, 2019 at 11:52 AM shane knapp wrote: > https://issues.apache.org/jira/browse/SPARK-26537 > > On Fri, Jan 4, 2019 at 11:31 AM shane knapp wrote: > >> this may push in to early next week... these builds were set up before >> my time, and i'm currently unraveling how they all work before pushing a >> commit to fix stuff. >> >> nothing like some code archaeology to make my friday more exciting! :) >> >> shane >> >> On Fri, Jan 4, 2019 at 11:08 AM Dongjoon Hyun >> wrote: >> >>> Thank you, Shane! >>> >>> Bests, >>> Dongjoon. >>> >>> On Fri, Jan 4, 2019 at 10:50 AM shane knapp wrote: >>> yeah, i'll get on that today. thanks for the heads up. On Fri, Jan 4, 2019 at 10:46 AM Dongjoon Hyun wrote: > Hi, All > > As a part of release process, we need to check Packaging/Compile/Test > Jenkins status. > > http://spark.apache.org/release-process.html > > 1. Spark Packaging: > https://amplab.cs.berkeley.edu/jenkins/view/Spark%20Packaging/ > 2. Spark QA Compile: > https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Compile/ > 3. Spark QA Test: > https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test/ > > Currently, (2) and (3) are working because it uses GitHub ( > https://github.com/apache/spark.git). > But, (1) seems to be broken because it's looking for old repo( > https://git-wip-us.apache.org/repos/asf/spark.git/info/refs) instead > of new GitBox. > > Can we fix this in this week? > > Bests, > Dongjoon. > > -- Shane Knapp UC Berkeley EECS Research / RISELab Staff Technical Lead https://rise.cs.berkeley.edu >>> >> >> -- >> Shane Knapp >> UC Berkeley EECS Research / RISELab Staff Technical Lead >> https://rise.cs.berkeley.edu >> > > > -- > Shane Knapp > UC Berkeley EECS Research / RISELab Staff Technical Lead > https://rise.cs.berkeley.edu >
Re: [DISCUSS] Handling correctness/data loss jiras
+1 Reynold Xin 于2019年1月4日周五 上午9:28写道： > Committers, > > When you merge tickets fixing correctness bugs, please make sure you tag > the tickets with "correctness" label. I've found multiple tickets today that > didn't do that. > > > On Fri, Aug 17, 2018 at 7:11 AM, Tom Graves > wrote: > >> Since we haven't heard any objections to this, the documentation has been >> updated (Thanks to Sean). >> >> All devs please make sure to re-read: >> http://spark.apache.org/contributing.html . >> >> Note the set of labels used in Jira has been documented and correctness >> or data loss issues should be marked as blocker by default. There is also >> a label to mark the jira as having something needing to go into the >> release-notes. >> >> >> Tom >> >> On Tuesday, August 14, 2018, 3:32:27 PM CDT, Imran Rashid < >> iras...@cloudera.com.INVALID> wrote: >> >> >> +1 on what we should do. >> >> On Mon, Aug 13, 2018 at 3:06 PM, Tom Graves > > wrote: >> >> >> > I mean, what are concrete steps beyond saying this is a problem? >> That's the important thing to discuss. >> >> Sorry I'm a bit confused by your statement but also think I agree. I >> started this thread for this reason. I pointed out that I thought it was a >> problem and also brought up things I thought we could do to help fix it. >> >> Maybe I wasn't clear in the first email, the list of things I had were >> proposals on what we do for a jira that is for a correctness/data loss >> issue. Its the committers and developers that are involved in this >> though so if people don't agree or aren't going to do them, then it doesn't >> work. >> >> Just to restate what I think we should do: >> >> - label any correctness/data loss jira with "correctness" >> - jira should be marked as a blocker by default if someone suspects a >> corruption/loss issue >> - Make sure the description is clear about when it occurs and impact to >> the user. >> - ensure its back ported to all active branches >> - See if we can have a separate section in the release notes for these >> >> The last one I guess is more a one time thing that i can file a jira >> for. The first 4 would be done for each jira filed. >> >> I'm proposing we do these things and as such if people agree we would >> also document those things in the committers or developers guide and send >> email to the list. >> >> >> >> Tom >> On Monday, August 13, 2018, 11:17:22 AM CDT, Sean Owen >> wrote: >> >> >> Generally: if someone thinks correctness fix X should be backported >> further, I'd say just do it, if it's to an active release branch (see >> below). Anything that important has to outweigh most any other concern, >> like behavior changes. >> >> >> On Mon, Aug 13, 2018 at 11:08 AM Tom Graves wrote: >> >> I'm not really sure what you mean by this, this proposal is to introduce >> a process for this type of issue so its at least brought to peoples >> attention. We can't do anything to make people work on certain things. If >> they aren't raised as important issues then its really easy to miss these >> things. If its a blocker we should also not be doing any new releases >> without a fix for it which may motivate people to look at it. >> >> >> I mean, what are concrete steps beyond saying this is a problem? That's >> the important thing to discuss. >> >> There's a good one here: let's say anything that's likely to be a >> correctness or data loss issue should automatically be labeled >> 'correctness' as such and set to Blocker. >> >> That can go into the how-to-contribute manual in the docs and in a note >> to dev@. >> >> >> >> I agree it would be good for us to make it more official about which >> branches are being maintained. I think at this point its still 2.1.x, >> 2.2.x, and 2.3.x since we recently did releases of all of these. Since 2.4 >> will be coming out we should definitely think about stop maintaining >> 2.1.x. Perhaps we need a table on our release page about this. But this >> should be a separate thread. >> >> >> I propose writing something like this in the 'versioning' doc page, to at >> least establish a policy: >> >> Minor release branches will, generally, be maintained with bug fixes >> releases for a period of 18 months. For example, branch 2.1.x is no longer >> considered maintained as of July 2018, 18 months after the release of 2.1.0 >> in December 2106. >> >> This gives us -- and more importantly users -- some understanding of what >> to expect for backporting and fixes. >> >> >> I am going to revive the thread about adding PMC / committers as it's >> overdue. That may not do much, but, more hands to do more work ought to >> possibly free up people to focus on deeper harder issues. >> >> >
Re: Spark Packaging Jenkins
IIRC there was a change to the release process: we stop using the shared gpg key on Jenkins, but use the personal key of the release manager. I'm not sure Jenkins can help testing package anymore. BTW release manager needs to run the packaging script by himself. If there is a problem, the release manager will find it out sooner or later. On Sun, Jan 6, 2019 at 6:34 AM Dongjoon Hyun wrote: > Hi, All. > > It turns out that `gpg signing` is the next huddle in Spark Packaging > Jenkins. > Since 2.4.0 release, is there something changed in our Jenkins machine? > > gpg: skipped > "/home/jenkins/workspace/spark-master-package/spark-utils/new-release-scripts/jenkins/jenkins-credentials-JEtz0nyn/gpg.tmp": > No secret key > gpg: signing failed: No secret key > > Bests, > Dongjoon. > > > On Fri, Jan 4, 2019 at 11:52 AM shane knapp wrote: > >> https://issues.apache.org/jira/browse/SPARK-26537 >> >> On Fri, Jan 4, 2019 at 11:31 AM shane knapp wrote: >> >>> this may push in to early next week... these builds were set up before >>> my time, and i'm currently unraveling how they all work before pushing a >>> commit to fix stuff. >>> >>> nothing like some code archaeology to make my friday more exciting! :) >>> >>> shane >>> >>> On Fri, Jan 4, 2019 at 11:08 AM Dongjoon Hyun >>> wrote: >>> Thank you, Shane! Bests, Dongjoon. On Fri, Jan 4, 2019 at 10:50 AM shane knapp wrote: > yeah, i'll get on that today. thanks for the heads up. > > On Fri, Jan 4, 2019 at 10:46 AM Dongjoon Hyun > wrote: > >> Hi, All >> >> As a part of release process, we need to check Packaging/Compile/Test >> Jenkins status. >> >> http://spark.apache.org/release-process.html >> >> 1. Spark Packaging: >> https://amplab.cs.berkeley.edu/jenkins/view/Spark%20Packaging/ >> 2. Spark QA Compile: >> https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Compile/ >> 3. Spark QA Test: >> https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test/ >> >> Currently, (2) and (3) are working because it uses GitHub ( >> https://github.com/apache/spark.git). >> But, (1) seems to be broken because it's looking for old repo( >> https://git-wip-us.apache.org/repos/asf/spark.git/info/refs) instead >> of new GitBox. >> >> Can we fix this in this week? >> >> Bests, >> Dongjoon. >> >> > > -- > Shane Knapp > UC Berkeley EECS Research / RISELab Staff Technical Lead > https://rise.cs.berkeley.edu > >>> >>> -- >>> Shane Knapp >>> UC Berkeley EECS Research / RISELab Staff Technical Lead >>> https://rise.cs.berkeley.edu >>> >> >> >> -- >> Shane Knapp >> UC Berkeley EECS Research / RISELab Staff Technical Lead >> https://rise.cs.berkeley.edu >> >
Re: Spark Packaging Jenkins
Thank you, Wenchen. I see. I'll update the doc and proceed to the next step manually as you advise. And it seems that we can stop the outdated Jenkins jobs, too. Bests, Dongjoon. On Sat, Jan 5, 2019 at 20:15 Wenchen Fan wrote: > IIRC there was a change to the release process: we stop using the shared > gpg key on Jenkins, but use the personal key of the release manager. I'm > not sure Jenkins can help testing package anymore. > > BTW release manager needs to run the packaging script by himself. If there > is a problem, the release manager will find it out sooner or later. > > > > On Sun, Jan 6, 2019 at 6:34 AM Dongjoon Hyun > wrote: > >> Hi, All. >> >> It turns out that `gpg signing` is the next huddle in Spark Packaging >> Jenkins. >> Since 2.4.0 release, is there something changed in our Jenkins machine? >> >> gpg: skipped >> "/home/jenkins/workspace/spark-master-package/spark-utils/new-release-scripts/jenkins/jenkins-credentials-JEtz0nyn/gpg.tmp": >> No secret key >> gpg: signing failed: No secret key >> >> Bests, >> Dongjoon. >> >> >> On Fri, Jan 4, 2019 at 11:52 AM shane knapp wrote: >> >>> https://issues.apache.org/jira/browse/SPARK-26537 >>> >>> On Fri, Jan 4, 2019 at 11:31 AM shane knapp wrote: >>> this may push in to early next week... these builds were set up before my time, and i'm currently unraveling how they all work before pushing a commit to fix stuff. nothing like some code archaeology to make my friday more exciting! :) shane On Fri, Jan 4, 2019 at 11:08 AM Dongjoon Hyun wrote: > Thank you, Shane! > > Bests, > Dongjoon. > > On Fri, Jan 4, 2019 at 10:50 AM shane knapp > wrote: > >> yeah, i'll get on that today. thanks for the heads up. >> >> On Fri, Jan 4, 2019 at 10:46 AM Dongjoon Hyun < >> dongjoon.h...@gmail.com> wrote: >> >>> Hi, All >>> >>> As a part of release process, we need to check >>> Packaging/Compile/Test Jenkins status. >>> >>> http://spark.apache.org/release-process.html >>> >>> 1. Spark Packaging: >>> https://amplab.cs.berkeley.edu/jenkins/view/Spark%20Packaging/ >>> 2. Spark QA Compile: >>> https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Compile/ >>> 3. Spark QA Test: >>> https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test/ >>> >>> Currently, (2) and (3) are working because it uses GitHub ( >>> https://github.com/apache/spark.git). >>> But, (1) seems to be broken because it's looking for old repo( >>> https://git-wip-us.apache.org/repos/asf/spark.git/info/refs) >>> instead of new GitBox. >>> >>> Can we fix this in this week? >>> >>> Bests, >>> Dongjoon. >>> >>> >> >> -- >> Shane Knapp >> UC Berkeley EECS Research / RISELab Staff Technical Lead >> https://rise.cs.berkeley.edu >> > -- Shane Knapp UC Berkeley EECS Research / RISELab Staff Technical Lead https://rise.cs.berkeley.edu >>> >>> >>> -- >>> Shane Knapp >>> UC Berkeley EECS Research / RISELab Staff Technical Lead >>> https://rise.cs.berkeley.edu >>> >>
