Hi, Krisztián, thanks for collecting the problems!
> *PREPARE_CHANGELOG* phase: > > - need to set ARROW_HOME because changelog.py requires it https://github.com/apache/arrow/pull/6975 > - changelog.py stopped working since adding support for parquet tickets [1], > because it requires the actual version to have a git tag, which is not yet > available during the release procedure (called from prepare.sh) We can test PREPARE_CHANGELOG phase by dev/release/00-prepare-test.rb if dev/release/changelog.py doesn't require JIRA account. > - on OSX brew installed ORC doesn't work so need to use the bundled source, > passing `-DORC_SOURCE=BUNDLED` fixes it I may take a look at this later. > - for me only OpenJDK 8 and Maven 3.5 version combination works, with newer > versions the build fails once a javadoc another time with an unknown error I opened https://issues.apache.org/jira/browse/ARROW-5764 for this about a year ago. But nobody works on this yet. Could any Java developer work on this? > [Packaging][deb] Support RC version numbers for apache-arrow-archive-keyring > [4] Sorry. I've fixed this: https://github.com/apache/arrow/commit/3e1680ed2a0d62ab3d86c32f53a942af4fefcc5d > [Packaging][rpm] Fix CentOS 6 build [5] Sorry again. I've fixed this too: https://github.com/apache/arrow/commit/37bf6e3a5b17da9c5328f173de2d6577921a429a Thanks, -- kou In <CAHM19a5wXT4ULVVesE+FPZ_4M=rah1mwofdi-sezwwh7hcs...@mail.gmail.com> "Thoughts and issues regarding the release procedure" on Fri, 17 Apr 2020 03:51:36 +0200, Krisztián Szűcs <szucs.kriszt...@gmail.com> wrote: > Hi, > > While our release scripts have improved a lot lately, cutting the first > release > candidate still takes multiple days. I wouldn't consider the overall > experience > bad - especially given the complexity of the project and the number of > artifacts we produce - but we definitely need to develop more automatisms and > tests supporting it. > I'm not sure what's the right way to have an action plan, but having more > manpower here would be great. > > If you don't mind, I'd like to specially thank Kou for maintaining most of the > release scripts and (when not being the RM) always helping out with the > upcoming issues, I really appreciate it. > > I tried to collect the problems, inconveniences I had with 0.17.0-RC0: > > 00-prepare.sh > ------------- > > *PREPARE_CHANGELOG* phase: > > - need to set ARROW_HOME because changelog.py requires it > - changelog.py stopped working since adding support for parquet tickets [1], > because it requires the actual version to have a git tag, which is not yet > available during the release procedure (called from prepare.sh) > > Extra *PREPARE_DEB_PACKAGE_NAMES* phase: > > This is usually not required but the previous `so` versions were set to .100, > so I had to downgrade them: > > ```bash > PREPARE_DEFAULT=0 PREPARE_DEB_PACKAGE_NAMES=1 \ > dev/release/00-prepare.sh 1.0.0 0.17.0 > ``` > > We should add this step to ensure that the so versions in the linux package > are properly set, and also consider to remove the previous version from the > pattern. > > *PREPARE_TAG* phase: > > The outstanding issue was the JNI ORC crash we have discussed on the mailing > list and I have a reproducer PR available for [6]. I had to `@Ignore` the > crashing test to be able to release. > > Minor issues: > > - on OSX brew installed ORC doesn't work so need to use the bundled source, > passing `-DORC_SOURCE=BUNDLED` fixes it > - need to update the maven versions to match <version>-SNAPSHOT with command > `mvn versions:set -DnewVersion=0.17.0-SNAPSHOT` > - gandiva has deprication warnings, which was complicating the debugging of > the java jni orc problem [2] > - for me only OpenJDK 8 and Maven 3.5 version combination works, with newer > versions the build fails once a javadoc another time with an unknown error > - if I rerun the script without removing the version tag then the maven > process raises an error *after* compiling everything again, effectively > losing 20 minutes. > > I'm generally frustrated about the compile time required iterate with the > maven > build issues - I suppose there are better ways to invoke maven which I'm not > aware of. It would be nice to have a Java developer guide listing the > recommended > commands in certain scenarios. > > Another thing I dislike about the release procedure is that the source code > tagging is done by / centered around maven. Preferably I would like to fire > the > `git tag` command explicitly rather than letting one of the many > package managers > to do it implicitly. > > 02-source.sh > ------------ > > The previous step produces a directory with the same name as the tag: > apache-arrow-0.17.0 which makes the script failing [3] > > Binary Packaging > ---------------- > > I had to apply two patches to fix the linux packaging builds. > > [Packaging][deb] Support RC version numbers for apache-arrow-archive-keyring > [4] > The packaging scripts were not properly supporting the -RC0 postfixed version > number which is a special case because the linux binaries are built agains > the apache source release rather than a git tag. While I managed to fix it, > we probably need a follow-up after the rebase. > > [Packaging][rpm] Fix CentOS 6 build [5] > This issue has surfaced today with the nightly builds as well, seems like > devtoolset-6 is no longer available for CentOS 6 so I had to update it. > > Building the 4 windows wheels on Appveyor takes 4 hours, because we don't have > any parallelism there. We should port the windows scripts to either Azure or > Github Actions. > > > Looking forward to the improvement ideas! > > Thanks Everyone! > > > [1] > https://github.com/apache/arrow/commit/636a912c4bef6803fe3fede8a050d82124b18136#diff-fc9c73b2cf4e254206ac116714cfdbf4 > [2] https://gist.github.com/kszucs/08b1582ca60a86c8dd8a1ab50bb6faad > [3] https://gist.github.com/kszucs/3337e475ce751cfbf11ea45a5a8817d2 > [4] > https://github.com/kszucs/arrow/commit/2c4cb4576a04b930a295ce6838179a8cf5a16058 > [5] > https://github.com/kszucs/arrow/commit/0b245aa3404bf016488e36e22a0140813b661f40 > [6] https://github.com/apache/arrow/pull/6953
