Hi Andy, Just curious with the next release coming up if you had a chance to look into the Maven issues yet.
Thanks, Micah On Thu, Aug 1, 2019 at 2:04 PM Wes McKinney <wesmck...@gmail.com> wrote: > I agree. In my experiences as RM I have found the involvement of Maven > in the release process to be a nuisance. I think it makes more sense > in Java-only projects > > On Thu, Aug 1, 2019 at 2:54 PM Andy Grove <andygrov...@gmail.com> wrote: > > > > I'll start taking a look at the maven issue. We might not want to use > maven > > release plugin given that we control the version number already across > this > > repository via other means. > > > > On Wed, Jul 31, 2019 at 4:26 PM Sutou Kouhei <k...@clear-code.com> wrote: > > > > > Hi, > > > > > > Sorry for not replying this thread. > > > > > > I think that the biggest problem is related to our Java > > > package. > > > > > > > > > We'll be able to resolve the GPG key problem by creating a > > > GPG key only for nightly release test. We can share the test > > > GPG key publicly because it's a just for testing. > > > > > > It'll work for our binary artifacts and APT/Yum repositories > > > but not work for our Java package. I don't know where GPG > > > key is used in our Java package... > > > > > > > > > We'll be able to resolve the Git commit problem by creating > > > a cloned Git repository for test. It's done in our > > > dev/release/00-prepare-test.rb[1]. > > > > > > [1] > > > > https://github.com/apache/arrow/blob/master/dev/release/00-prepare-test.rb#L30 > > > > > > The biggest problem for the Git commit is our Java package > > > requires "apache-arrow-${VERSION}" tag on > > > https://github.com/apache/arrow . (Right?) > > > I think that "mvm release:perform" in > > > dev/release/01-perform.sh does so but I don't know the > > > details of "mvm release:perform"... > > > > > > > > > More details: > > > > > > dev/release/00-prepare.sh: > > > > > > We'll be able to run this automatically when we can resolve > > > the above GPG key problem in our Java package. We can > > > resolve the Git commit problem by creating a cloned Git > > > repository. > > > > > > dev/release/01-prepare.sh: > > > > > > We'll be able to run this automatically when we can resolve > > > the above Git commit ("apche-arrow-${VERSION}" tag) problem > > > in our Java package. > > > > > > dev/release/02-source.sh: > > > > > > We'll be able to run this automatically by creating a GPG > > > key for nightly release test. We'll use Bintray to upload RC > > > source archive instead of dist.apache.org. Ah, we need a > > > Bintray API key for this. It must be secret. > > > > > > dev/release/03-binary.sh: > > > > > > We'll be able to run this automatically by creating a GPG > > > key for nightly release test. We need a Bintray API key too. > > > > > > We need to improve this to support nightly release test. It > > > use "XXX-rc" such as "debian-rc" for Bintray "package" name. > > > It should use "XXX-nightly" such as "debian-nightly" for > > > nightly release test instead. > > > > > > dev/release/post-00-release.sh: > > > > > > We'll be able to skip this. > > > > > > dev/release/post-01-upload.sh: > > > > > > We'll be able to skip this. > > > > > > dev/release/post-02-binary.sh: > > > > > > We'll be able to run this automatically by creating Bintray > > > "packages" for nightly release and use them. We can create > > > "XXX-nightly-release" ("debian-nightly-release") Bintray > > > "packages" and use them instead of "XXX" ("debian") Bintray > > > "packages". > > > > > > "debian" Bintray "package": https://bintray.com/apache/debian/ > > > > > > We need to improve this to support nightly release. > > > > > > dev/release/post-03-website.sh: > > > > > > We'll be able to run this automatically by creating a cloned > > > Git repository for test. > > > > > > It's better that we have a Web site to show generated pages. > > > We can create > > > https://github.com/apache/arrow-site/tree/asf-site/nightly > > > and use it but I don't like it. Because arrow-site increases > > > a commit day by day. > > > Can we prepare a Web site for this? (arrow-nightly.ursalabs.org?) > > > > > > dev/release/post-04-rubygems.sh: > > > > > > We may be able to use GitHub Package Registry[2] to upload > > > RubyGems. We can use "pre-release" package feature of > > > https://rubygems.org/ but it's not suitable for > > > nightly. It's for RC or beta release. > > > > > > [2] > https://github.blog/2019-05-10-introducing-github-package-registry/ > > > > > > dev/release/post-05-js.sh: > > > > > > We may be able to use GitHub Package Registry[2] to upload > > > npm packages. > > > > > > dev/release/post-06-csharp.sh: > > > > > > We may be able to use GitHub Package Registry[2] to upload > > > NuGet packages. > > > > > > dev/release/post-07-rust.sh: > > > > > > I don't have any idea. But it must be ran > > > automatically. It's always failed. I needed to run each > > > command manually. > > > > > > dev/release/post-08-remove-rc.sh: > > > > > > We'll be able to skip this. > > > > > > > > > Thanks, > > > -- > > > kou > > > > > > In <CAJPUwMAmp0jvz6qrdqehBEUB_NdbGEsHFNgLW9Q6V9RFTnk=3...@mail.gmail.com > > > > > "Re: [DISCUSS] Release cadence and release vote conventions" on Wed, > 31 > > > Jul 2019 15:35:57 -0500, > > > Wes McKinney <wesmck...@gmail.com> wrote: > > > > > > > The PMC member and their GPG keys need to be in the loop at some > > > > point. The release artifacts can be produced by some kind of CI/CD > > > > system so long as the PMC member has confidence in the security of > > > > those artifacts before signing them. For example, we build the > > > > official binary packages on public CI services and then download and > > > > sign them with Crossbow. I think the same could be done in theory > with > > > > the source release but we'd first need to figure out what to do about > > > > the parts that create git commits. > > > > > > > > On Wed, Jul 31, 2019 at 11:23 AM Andy Grove <andygrov...@gmail.com> > > > wrote: > > > >> > > > >> To what extent would it be possible to automate the release process > via > > > >> CICD? > > > >> > > > >> On Wed, Jul 31, 2019 at 9:19 AM Wes McKinney <wesmck...@gmail.com> > > > wrote: > > > >> > > > >> > I think one thing that would help would be improving the > > > >> > reproducibility of the source release process. The RM has to have > > > >> > their machine configured in a particular way for it to work. > > > >> > > > > >> > Before anyone says "Docker" it isn't an easy solution because the > > > >> > release scripts need to be able to create git commits (created by > the > > > >> > Maven release plugin) and sign artifacts using the RM's GPG keys. > > > >> > > > > >> > On Sat, Jul 27, 2019 at 10:04 PM Micah Kornfield < > > > emkornfi...@gmail.com> > > > >> > wrote: > > > >> > > > > > >> > > I just wanted to bump this thread. Kou and Krisztián as the > last > > > two > > > >> > > release managers is there any specific infrastructure that you > think > > > >> > might > > > >> > > have helped? > > > >> > > > > > >> > > Thanks, > > > >> > > Micah > > > >> > > > > > >> > > On Wed, Jul 17, 2019 at 11:29 PM Micah Kornfield < > > > emkornfi...@gmail.com> > > > >> > > wrote: > > > >> > > > > > >> > > > I'd can help as well, but not exactly sure where to start. It > > > seems > > > >> > like > > > >> > > > there are already some JIRAs opened [1] > > > >> > > > for improving the release? Could someone more familiar with > the > > > >> > process > > > >> > > > pick out the highest priority ones? Do more need to be opened? > > > >> > > > > > > >> > > > Thanks, > > > >> > > > Micah > > > >> > > > > > > >> > > > [1] > > > >> > > > > > > >> > > > > > https://issues.apache.org/jira/browse/ARROW-2880?jql=project%20%3D%20ARROW%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20component%20in%20(%22Developer%20Tools%22%2C%20Packaging)%20and%20summary%20~%20Release > > > >> > > > > > > >> > > > On Sat, Jul 13, 2019 at 7:17 AM Wes McKinney < > wesmck...@gmail.com > > > > > > > >> > wrote: > > > >> > > > > > > >> > > >> To be effective at improving the life of release managers, > the > > > nightly > > > >> > > >> release process really should use as close as possible to the > > > same > > > >> > > >> scripts that the RM uses to produce the release. Otherwise we > > > could > > > >> > > >> have a situation where the nightlies succeed but there is > some > > > problem > > > >> > > >> that either fails an RC or is unable to be produced at all. > > > >> > > >> > > > >> > > >> On Sat, Jul 13, 2019 at 9:12 AM Andy Grove < > > > andygrov...@gmail.com> > > > >> > wrote: > > > >> > > >> > > > > >> > > >> > I would like to volunteer to help with Java and Rust > release > > > process > > > >> > > >> work, > > > >> > > >> > especially nightly releases. > > > >> > > >> > > > > >> > > >> > Although I'm not that familiar with the Java > implementation of > > > >> > Arrow, I > > > >> > > >> > have been using Java and Maven for a very long time. > > > >> > > >> > > > > >> > > >> > Do we envisage a single nightly release process that > releases > > > all > > > >> > > >> languages > > > >> > > >> > simultaneously? or do we want separate process per > language, > > > with > > > >> > > >> different > > > >> > > >> > maintainers? > > > >> > > >> > > > > >> > > >> > > > > >> > > >> > > > > >> > > >> > On Wed, Jul 10, 2019 at 8:18 AM Wes McKinney < > > > wesmck...@gmail.com> > > > >> > > >> wrote: > > > >> > > >> > > > > >> > > >> > > On Sun, Jul 7, 2019 at 7:40 PM Sutou Kouhei < > > > k...@clear-code.com> > > > >> > > >> wrote: > > > >> > > >> > > > > > > >> > > >> > > > Hi, > > > >> > > >> > > > > > > >> > > >> > > > > in future releases we should > > > >> > > >> > > > > institute a minimum 24-hour "quiet period" after any > > > community > > > >> > > >> > > > > feedback on a release candidate to allow issues to be > > > examined > > > >> > > >> > > > > further. > > > >> > > >> > > > > > > >> > > >> > > > I agree with this. I'll do so when I do a release > manager > > > in > > > >> > > >> > > > the future. > > > >> > > >> > > > > > > >> > > >> > > > > To be able to release more often, two things have to > > > happen: > > > >> > > >> > > > > > > > >> > > >> > > > > * More PMC members must engage with the release > > > management > > > >> > role, > > > >> > > >> > > > > process, and tools > > > >> > > >> > > > > * Continued improvements to release tooling to make > the > > > >> > process > > > >> > > >> less > > > >> > > >> > > > > painful for the release manager. For example, it > seems > > > we may > > > >> > > >> want to > > > >> > > >> > > > > find a different place than Bintray to host binary > > > artifacts > > > >> > > >> > > > > temporarily during release votes > > > >> > > >> > > > > > > >> > > >> > > > My opinion that we need to build nightly release > system. > > > >> > > >> > > > > > > >> > > >> > > > It uses dev/release/NN-*.sh to build .tar.gz and binary > > > >> > > >> > > > artifacts from the .tar.gz. > > > >> > > >> > > > It also uses dev/release/verify-release-candidate.* to > > > >> > > >> > > > verify build .tar.gz and binary artifacts. > > > >> > > >> > > > It also uses dev/release/post-NN-*.sh to do post > release > > > >> > > >> > > > tasks. (Some tasks such as uploading a package to > packaging > > > >> > > >> > > > system will be dry-run.) > > > >> > > >> > > > > > > >> > > >> > > > > > >> > > >> > > I agree that having a turn-key release system that's > capable > > > of > > > >> > > >> > > producing nightly packages is the way to do. That way any > > > problems > > > >> > > >> > > that would block a release will come up as they happen > > > rather than > > > >> > > >> > > piling up until the very end like they are now. > > > >> > > >> > > > > > >> > > >> > > > I needed 10 or more changes for dev/release/ to create > > > >> > > >> > > > 0.14.0 RC0. (Some of them are still in my local > stashes. I > > > >> > > >> > > > don't have time to create pull requests for them > > > >> > > >> > > > yet. Because I postponed some tasks of my main > > > >> > > >> > > > business. I'll create pull requests after I finished > the > > > >> > > >> > > > postponed tasks of my main business.) > > > >> > > >> > > > > > > >> > > >> > > > > > >> > > >> > > Thanks. I'll follow up on the 0.14.1/0.15.0 thread -- > since > > > we > > > >> > need to > > > >> > > >> > > release again soon because of problems with 0.14.0 please > > > let us > > > >> > know > > > >> > > >> > > what patches will be needed to make another release. > > > >> > > >> > > > > > >> > > >> > > > If we fix problems related to dev/release/ in our > normal > > > >> > > >> > > > development process, release process will be less > painful. > > > >> > > >> > > > > > > >> > > >> > > > The biggest problem for 0.14.0 RC0 is java/pom.xml > related: > > > >> > > >> > > > https://github.com/apache/arrow/pull/4717 > > > >> > > >> > > > > > > >> > > >> > > > It was difficult for me because I don't have Java > > > >> > > >> > > > knowledge. Release manager needs help from many > developers > > > >> > > >> > > > because release manager may not have knowledge of all > > > >> > > >> > > > supported languages. Apache Arrow supports 10 over > > > >> > > >> > > > languages. > > > >> > > >> > > > > > > >> > > >> > > > > > > >> > > >> > > > For Bintray API limit problem, we'll be able to > resolve it. > > > >> > > >> > > > I was added to https://bintray.com/apache/ members: > > > >> > > >> > > > > > > >> > > >> > > > https://issues.apache.org/jira/browse/INFRA-18698 > > > >> > > >> > > > > > > >> > > >> > > > I'll be able to use Bintray API without limitation in > the > > > >> > > >> > > > future. Release managers should also request the same > > > thing. > > > >> > > >> > > > > > > >> > > >> > > > > > >> > > >> > > This is good, I will add myself. Other PMC members should > > > also add > > > >> > > >> > > themselves. > > > >> > > >> > > > > > >> > > >> > > > > > > >> > > >> > > > Thanks, > > > >> > > >> > > > -- > > > >> > > >> > > > kou > > > >> > > >> > > > > > > >> > > >> > > > In <CAJPUwMBRzYQ=hbVwFuPYAB-O= > > > >> > > >> lsowxqxidjapc_cofguksj...@mail.gmail.com> > > > >> > > >> > > > "[DISCUSS] Release cadence and release vote > conventions" > > > on > > > >> > Sat, > > > >> > > >> 6 Jul > > > >> > > >> > > 2019 16:28:50 -0500, > > > >> > > >> > > > Wes McKinney <wesmck...@gmail.com> wrote: > > > >> > > >> > > > > > > >> > > >> > > > > hi folks, > > > >> > > >> > > > > > > > >> > > >> > > > > As a reminder, particularly since we have many new > > > community > > > >> > > >> members > > > >> > > >> > > > > (some of whom have never been involved with an ASF > > > project > > > >> > > >> before), > > > >> > > >> > > > > releases are approved exclusively by the PMC and in > > > general > > > >> > > >> releases > > > >> > > >> > > > > cannot be vetoed. In spite of that, we strive to make > > > releases > > > >> > > >> that > > > >> > > >> > > > > have unanimous (either by explicit +1 or lazy > consent) > > > >> > support of > > > >> > > >> the > > > >> > > >> > > > > PMC. So it is better to have unanimous 5 +1 votes > than 6 > > > +1 > > > >> > votes > > > >> > > >> with > > > >> > > >> > > > > a -1 dissenting vote. > > > >> > > >> > > > > > > > >> > > >> > > > > On the 0.14.0 vote, as with previous release votes, > some > > > >> > issues > > > >> > > >> with > > > >> > > >> > > > > the release were raised by members of the community, > > > whether > > > >> > > >> build or > > > >> > > >> > > > > test-related problems or other failures. Technically > > > speaking, > > > >> > > >> such > > > >> > > >> > > > > issues have no _direct_ bearing on whether a release > vote > > > >> > passes, > > > >> > > >> only > > > >> > > >> > > > > on whether PMC members vote +1, 0, or -1. A PMC > member is > > > >> > allowed > > > >> > > >> to > > > >> > > >> > > > > change their vote based on new information -- for > > > example, if > > > >> > I > > > >> > > >> voted > > > >> > > >> > > > > +1 on a release and then someone reported a serious > > > licensing > > > >> > > >> issue, > > > >> > > >> > > > > then I would revise my vote to -1. > > > >> > > >> > > > > > > > >> > > >> > > > > On the RC0 vote thread, Jacques wrote [1] > > > >> > > >> > > > > > > > >> > > >> > > > > "A release vote should last until we arrive at > consensus. > > > >> > When an > > > >> > > >> > > > > issue is potentially identified, those that have > voted > > > should > > > >> > be > > > >> > > >> given > > > >> > > >> > > > > ample time to change their vote and others that may > have > > > been > > > >> > lazy > > > >> > > >> > > > > consenters should be given time to chime in. There > is no > > > >> > maximum > > > >> > > >> > > > > amount of time a vote can be open. Allowing at least > 24 > > > hours > > > >> > > >> after an > > > >> > > >> > > > > objection is raised is a pretty minimum expectation > > > unless the > > > >> > > >> > > > > objector removes their objection. > > > >> > > >> > > > > > > > >> > > >> > > > > Note that Apache is more focused on consensus than > > > timing (as > > > >> > > >> opposed > > > >> > > >> > > to > > > >> > > >> > > > > virtually other other organizations in the world)." > > > >> > > >> > > > > > > > >> > > >> > > > > I agree with this and my opinion is that in future > > > releases we > > > >> > > >> should > > > >> > > >> > > > > institute a minimum 24-hour "quiet period" after any > > > community > > > >> > > >> > > > > feedback on a release candidate to allow issues to be > > > examined > > > >> > > >> > > > > further. If someone finds a potential problem, and no > > > negative > > > >> > > >> votes > > > >> > > >> > > > > are cast or changed, then the vote can close. > > > >> > > >> > > > > > > > >> > > >> > > > > As a related matter, it seems clear to me that Apache > > > Arrow > > > >> > should > > > >> > > >> > > > > have more frequent releases. I think this would > decrease > > > >> > pressure > > > >> > > >> on > > > >> > > >> > > > > developers and users alike. While we've made strides > to > > > >> > improve > > > >> > > >> the > > > >> > > >> > > > > tooling for release management (big thanks to Kou, > > > Yosuke, > > > >> > > >> Krisztian, > > > >> > > >> > > > > and others), there is still quite some labor > involved and > > > >> > > >> potential > > > >> > > >> > > > > for issues (e.g. API rate limiting for binary > artifacts > > > on > > > >> > > >> Bintray). > > > >> > > >> > > > > > > > >> > > >> > > > > To be able to release more often, two things have to > > > happen: > > > >> > > >> > > > > > > > >> > > >> > > > > * More PMC members must engage with the release > > > management > > > >> > role, > > > >> > > >> > > > > process, and tools > > > >> > > >> > > > > * Continued improvements to release tooling to make > the > > > >> > process > > > >> > > >> less > > > >> > > >> > > > > painful for the release manager. For example, it > seems > > > we may > > > >> > > >> want to > > > >> > > >> > > > > find a different place than Bintray to host binary > > > artifacts > > > >> > > >> > > > > temporarily during release votes > > > >> > > >> > > > > > > > >> > > >> > > > > Any other ideas for things we can do to improve the > > > process > > > >> > and > > > >> > > >> > > > > cadence of releases? > > > >> > > >> > > > > > > > >> > > >> > > > > Thanks, > > > >> > > >> > > > > Wes > > > >> > > >> > > > > > > > >> > > >> > > > > [1]: > > > >> > > >> > > > > > >> > > >> > > > >> > > > > > https://lists.apache.org/thread.html/be6210e97b838494a5516dad6408f479efe4c98aff805000597c0196@%3Cdev.arrow.apache.org%3E > > > >> > > >> > > > > > >> > > >> > > > >> > > > > > > >> > > > > >
