Vote summary: +1 Xiaoqiao He (binding) Steve (binding) Ahmar (binding) Wei-Chiu Chuang (binding) Mukund (binding) Syed Shameerur Rahman (non-binding) Cheng Pan (non-binding) Dongjoon Hyun (non-binding) Sneha Vijayarajan (non-binding)
=0 Ayush Saxena -1 None With 5 binding +1 votes and 4 non-binding +1 votes, voting has passed. We did verify the problem with the ARM release which Ayush found and thus we have decided to not put the arm binaries in this release. Thanks everyone for all the help during the release process. Release announcement to follow soon! On Fri, Oct 18, 2024 at 4:44 PM Mukund Madhav Thakur <mtha...@cloudera.com> wrote: > Thanks Ayush for finding this issue. Looks like there is a problem with > the ARM binaries. The reason for that could be that the create-release > script doesn't work in ARM docker container but works manually after > logging inside the container. So I was creating the tars manually. Seems > like it didn't work properly. > There are two yarn issues because of which I had to create this manually. > https://issues.apache.org/jira/browse/YARN-11712 > https://issues.apache.org/jira/browse/YARN-11713 > So we have decided to remove the aarch* binaries from the 3.4.1 release > and target to fix that in 3.4.2. > > > > > On Fri, Oct 18, 2024 at 1:41 PM Ayush Saxena <ayush...@gmail.com> wrote: > >> Thanx Mukund for sharing the details, I might have missed that >> discussion. Do mention the purpose on the website once we upload the >> lean jar, so that relevant people can use it. >> >> I am holding my vote due to [3], that in general looks blocker to me, >> If I haven't messed up. Would be great if someone can help validate >> that I believe is the cause for [2] >> >> * Built from source >> * Verified checksums >> * Verified signatures >> * Validated no code diff b/w the git tag & src tar >> * Verified the output of `hadoop version` [1] >> * Checked the NOTICE & LICENSE file >> * Ran some basic HDFS shell commands >> * Ran some Erasure Coding related commands >> * Ran example jobs (TeraGen, TeraSort, WordCount) [2] >> * Browsed through the UI (NN, DN, RM, NM & JHS) >> * Skimmed over the contents of maven artifacts, Release Notes & ChangeLog >> * Discrepancy in contents of aarch64 tar & x86 tar [3] >> >> [1] >> >> The hadoop version output inside the aarch64 tar shows the source >> repository as Unknown, which isn't the case in other tar nor was the >> case in the last 3.4.0 release. Not blocking, but we should figure out >> & fix it in the next release. >> >> ``` >> ayushsaxena@ayushsaxena hadoop-3.4.1 % bin/hadoop version >> Hadoop 3.4.1 >> Source code repository Unknown -r Unknown >> Compiled by mthakur on 2024-10-10T08:24Z >> Compiled on platform linux-aarch_64 >> Compiled with protoc 3.23.4 >> From source with checksum 7292fe9dba5e2e44e3a9f763fce3e680 >> ``` >> >> [2] While trying to start ResourceManager via the ARM binary, it is >> failing for me: >> ``` >> ayushsaxena@ayushsaxena hadoop-3.4.1 % bin/yarn resourcemanager >> Error: Could not find or load main class >> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager >> ``` >> It didn't used to happen earlier, but I sorted it out by "export >> >> HADOOP_CLASSPATH=$HADOOP_CLASSPATH:/Users/ayushsaxena/code/RC/hadoop-3.4.1/hadoop-3.4.1/share/hadoop/tools/lib/*", >> Considering I am not pro with YARN, I might have messed up the setup. >> My mapred settings have Tez related stuff as well, but the same works >> if I use the x86 binary.... >> >> >> [3] Investigating [2], I think the YARN Jars are missing in the >> aarch64 share/hadoop/yarn & they are there in the share/hadoop/tools >> directory >> >> in x86 >> ``` >> ayushsaxena@ayushsaxena hadoop-3.4.1 % ls >> x86/hadoop-3.4.1/share/hadoop/yarn >> csi hadoop-yarn-server-router-3.4.1.jar >> hadoop-yarn-api-3.4.1.jar hadoop-yarn-server-sharedcachemanager-3.4.1.jar >> hadoop-yarn-applications-catalog-webapp-3.4.1.war >> hadoop-yarn-server-tests-3.4.1.jar >> hadoop-yarn-applications-distributedshell-3.4.1.jar >> hadoop-yarn-server-timeline-pluginstorage-3.4.1.jar >> hadoop-yarn-applications-mawo-core-3.4.1.jar >> hadoop-yarn-server-web-proxy-3.4.1.jar >> hadoop-yarn-applications-unmanaged-am-launcher-3.4.1.jar >> hadoop-yarn-services-api-3.4.1.jar >> hadoop-yarn-client-3.4.1.jar hadoop-yarn-services-core-3.4.1.jar >> hadoop-yarn-common-3.4.1.jar lib >> hadoop-yarn-registry-3.4.1.jar sources >> hadoop-yarn-server-applicationhistoryservice-3.4.1.jar test >> hadoop-yarn-server-common-3.4.1.jar timelineservice >> hadoop-yarn-server-globalpolicygenerator-3.4.1.jar webapps >> hadoop-yarn-server-nodemanager-3.4.1.jar yarn-service-examples >> hadoop-yarn-server-resourcemanager-3.4.1.jar >> ayushsaxena@ayushsaxena hadoop-3.4.1 % >> >> ``` >> >> And in aarch64 they aren't there >> >> ``` >> >> ayushsaxena@ayushsaxena hadoop-3.4.1 % ls -l >> aarch/hadoop-3.4.1/share/hadoop/yarn >> >> total 0 >> drwxr-xr-x 3 ayushsaxena staff 96 Oct 10 15:34 csi >> drwxr-xr-x 47 ayushsaxena staff 1504 Oct 10 15:34 lib >> drwxr-xr-x 28 ayushsaxena staff 896 Oct 10 15:34 sources >> drwxr-xr-x 3 ayushsaxena staff 96 Oct 10 15:34 timelineservice >> drwxr-xr-x 3 ayushsaxena staff 96 Oct 10 15:34 webapps >> drwxr-xr-x 6 ayushsaxena staff 192 Oct 10 15:34 yarn-service-examples >> ayushsaxena@ayushsaxena hadoop-3.4.1 % >> ``` >> >> Can someone help double check this once? I downloaded the 3.4.0 binary >> & this is not the case over there. I did a build locally on aarch64 >> and there also the jars are present in the share/hadoop/yarn dir >> >> -Ayush >> >> On Fri, 18 Oct 2024 at 11:57, Mukund Madhav Thakur <mtha...@cloudera.com> >> wrote: >> > >> > Hi Ayush, >> > "lean" tar is a small tar file which doesn't contain the AWS SDK >> because the size of AWS SDK is itself 500 MB. >> > This can ease usage for non AWS users. Even AWS users can add this jar >> explicitly if desired. >> > >> > This is created using https://github.com/apache/hadoop-release-support >> which is the new and easy way to create and validate hadoop releases >> > created by Steve during 3.4.0. Also the link to this is present in [1] . >> > There was a discussion around this during the RC2 testing. >> > >> > I will check with Steve on why we don't have the equivalent for aarch64. >> > >> > [1] https://cwiki.apache.org/confluence/display/HADOOP2/HowToRelease >> > >> > On Thu, Oct 17, 2024 at 10:00 PM Ayush Saxena <ayush...@gmail.com> >> wrote: >> >> >> >> What exactly is this "lean tar"? I couldn’t find any mention of it in >> >> [1], nor did I come across any thread establishing consensus on adding >> >> it, as we did for the aarch64 tar. It’s also not clear to me from the >> >> create-release script how it is getting generated. So, where is this >> >> coming from? Additionally, why don’t we have an equivalent for >> >> aarch64? >> >> >> >> -Ayush >> >> >> >> [1] https://cwiki.apache.org/confluence/display/HADOOP2/HowToRelease >> >> >> >> On Thu, 17 Oct 2024 at 13:21, Mukund Madhav Thakur >> >> <mtha...@cloudera.com.invalid> wrote: >> >> > >> >> > Adding my +1 (binding) after all the testing. >> >> > Ran a lot of validation workflows from hadoop-release-support all >> went >> >> > fine. >> >> > Verified checksum and signatures. >> >> > Build orc and reran the failing vectored io tests and it succeeded. >> >> > Build parquet and run the hadoop parquet module tests. >> >> > @Steve Loughran <ste...@apache.org> Build gcs and ran all the >> tests which >> >> > went fine. >> >> > >> >> > Also Ahmar is a committer so his vote can be counted as +1 as well. >> >> > >> >> > Thanks everyone for testing! Need just one more binding vote for >> this to be >> >> > released. >> >> > >> >> > >> >> > >> >> > >> >> > >> >> > On Thu, Oct 17, 2024 at 10:01 AM Sneha Vijayarajan < >> >> > sneha.vijayara...@gmail.com> wrote: >> >> > >> >> > > +1 (non-binding) >> >> > > >> >> > > Ran the ABFS driver tests and all the tests passed. >> >> > > >> >> > > ============================================================ >> >> > > HNS-OAuth >> >> > > ============================================================ >> >> > > >> >> > > [WARNING] Tests run: 154, Failures: 0, Errors: 0, Skipped: 2 >> >> > > [WARNING] Tests run: 646, Failures: 0, Errors: 0, Skipped: 82 >> >> > > [WARNING] Tests run: 433, Failures: 0, Errors: 0, Skipped: 57 >> >> > > >> >> > > ============================================================ >> >> > > HNS-SharedKey >> >> > > ============================================================ >> >> > > >> >> > > [WARNING] Tests run: 154, Failures: 0, Errors: 0, Skipped: 3 >> >> > > [WARNING] Tests run: 646, Failures: 0, Errors: 0, Skipped: 34 >> >> > > [WARNING] Tests run: 433, Failures: 0, Errors: 0, Skipped: 44 >> >> > > >> >> > > ============================================================ >> >> > > NonHNS-SharedKey >> >> > > ============================================================ >> >> > > >> >> > > [WARNING] Tests run: 154, Failures: 0, Errors: 0, Skipped: 9 >> >> > > [WARNING] Tests run: 630, Failures: 0, Errors: 0, Skipped: 274 >> >> > > [WARNING] Tests run: 433, Failures: 0, Errors: 0, Skipped: 47 >> >> > > >> >> > > ============================================================ >> >> > > AppendBlob-HNS-OAuth >> >> > > ============================================================ >> >> > > >> >> > > [WARNING] Tests run: 154, Failures: 0, Errors: 0, Skipped: 2 >> >> > > [WARNING] Tests run: 646, Failures: 0, Errors: 0, Skipped: 84 >> >> > > [WARNING] Tests run: 433, Failures: 0, Errors: 0, Skipped: 81 >> >> > > >> >> > > Thanks, >> >> > > Sneha Vijayarajan >> >> > > >> >> > > On Thu, Oct 17, 2024 at 8:57 AM Xiaoqiao He <hexiaoq...@apache.org> >> wrote: >> >> > > > >> >> > > > Update my vote to +1(binding). >> >> > > > >> >> > > > Addendum: Verified signature was correct for both src/binary/site >> >> > > tarball. >> >> > > > >> >> > > > >> >> > > > On Mon, Oct 14, 2024 at 5:26 PM Xiaoqiao He < >> hexiaoq...@apache.org> >> >> > > wrote: >> >> > > > >> >> > > > > Thanks Mukund and Steve for driving this release. >> >> > > > > >> >> > > > > +0. Will +1 when signature check passed. >> >> > > > > >> >> > > > > [Y] LICENSE files exist and NOTICE is included. >> >> > > > > [Y] Rat check is ok. mvn clean apache-rat:check >> >> > > > > [Y] Build the source code on Ubuntu and OpenJDK 11 by `mvn >> clean >> >> > > package >> >> > > > > -DskipTests -Pnative -Pdist -Dtar`. >> >> > > > > [Y] Setup pseudo cluster with HDFS and YARN. >> >> > > > > [Y] Run simple FsShell - mkdir/put/get/mv/rm and check the >> result. >> >> > > > > [Y] Run example mr jobs and check the result - Pi & wordcount. >> >> > > > > [Y] Spot-check and run some unit tests. >> >> > > > > [Y] Skimmed the Web UI of >> NameNode/DataNode/Resourcemanager/NodeManager >> >> > > > > etc. >> >> > > > > [Y] Skimmed over the contents of site documentation. >> >> > > > > [Y] Skimmed over the contents of maven repo. >> >> > > > > [N] Verified checksum passed but signature didn't pass. Did >> you use >> >> > > > > another key to sign? >> >> > > > > >> >> > > > >> hexiaoqiao@hexiaoqiao-ubuntu:~/Public/hadoop-3.4.1$ gpg >> --list-key | >> >> > > > >> grep -B2 -A1 "Mukund" >> >> > > > >> pub rsa4096 2023-09-28 [SC] [expires: 2027-09-28] >> >> > > > >> 41C7034C031FB3AF4BBFB5B89F070EBA4202CEC1 >> >> > > > >> uid [ unknown] Mukund Thakur <mtha...@apache.org> >> >> > > > >> sub rsa4096 2023-09-28 [E] [expires: 2027-09-28] >> >> > > > >> hexiaoqiao@hexiaoqiao-ubuntu:~/Public/hadoop-3.4.1$ gpg >> --verify >> >> > > > >> hadoop-3.4.1.tar.gz.asc hadoop-3.4.1.tar.gz >> >> > > > >> gpg: Signature made 2024年10月10日 星期四 01时10分30秒 CST >> >> > > > >> gpg: using RSA key >> >> > > 53931DAA708291409958BD474D22BB7D32882201 >> >> > > > >> gpg: Can't check signature: No public key >> >> > > > > >> >> > > > > >> >> > > > > Best Regards, >> >> > > > > - He Xiaoqiao >> >> > > > > >> >> > > > > On Fri, Oct 11, 2024 at 1:31 AM Steve Loughran >> >> > > <ste...@cloudera.com.invalid> >> >> > > > > wrote: >> >> > > > > >> >> > > > >> Hney, you did the -lean one too! nice >> >> > > > >> >> >> > > > >> Anyway, yes I'll test. I will try not to find new problems >> >> > > > >> >> >> > > > >> >> >> > > > >> On Thu, 10 Oct 2024 at 12:18, Mukund Madhav Thakur >> >> > > > >> <mtha...@cloudera.com.invalid> wrote: >> >> > > > >> >> >> > > > >> > Apache Hadoop 3.4.1 >> >> > > > >> > >> >> > > > >> > >> >> > > > >> > With help of Steve, I have put together a release candidate >> (RC3) >> >> > > for >> >> > > > >> > Hadoop 3.4.1. >> >> > > > >> > >> >> > > > >> > >> >> > > > >> > What we would like is for anyone who can to verify the >> tarballs, >> >> > > > >> especially >> >> > > > >> > >> >> > > > >> > anyone who can try the arm64 binaries as we want to include >> them >> >> > > too. >> >> > > > >> > >> >> > > > >> > >> >> > > > >> > The RC is available at: >> >> > > > >> > >> >> > > > >> > >> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.1-RC3/ >> >> > > > >> > >> >> > > > >> > >> >> > > > >> > The git tag is release-3.4.1-RC3, commit >> >> > > > >> > 4d7825309348956336b8f06a08322b78422849b1 >> >> > > > >> > >> >> > > > >> > >> >> > > > >> > The maven artifacts are staged at >> >> > > > >> > >> >> > > > >> > >> >> > > >> https://repository.apache.org/content/repositories/orgapachehadoop-1430 >> >> > > > >> > >> >> > > > >> > >> >> > > > >> > You can find my public key at: >> >> > > > >> > >> >> > > > >> > >> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS >> >> > > > >> > >> >> > > > >> > >> >> > > > >> > Change log >> >> > > > >> > >> >> > > > >> > >> >> > > > >> >> >> > > >> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.1-RC3/CHANGELOG.md >> >> > > > >> > >> >> > > > >> > >> >> > > > >> > Release notes >> >> > > > >> > >> >> > > > >> > >> >> > > > >> > >> >> > > > >> >> >> > > >> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.1-RC3/RELEASENOTES.md >> >> > > > >> > >> >> > > > >> > >> >> > > > >> > This is off branch-3.4.1 >> >> > > > >> > >> >> > > > >> > >> >> > > > >> > Key changes include >> >> > > > >> > >> >> > > > >> > >> >> > > > >> > * Bulk Delete API. >> >> > > https://issues.apache.org/jira/browse/HADOOP-18679 >> >> > > > >> > >> >> > > > >> > * Fixes and enhancements in Vectored IO API. >> >> > > > >> > >> >> > > > >> > * Improvements in Hadoop Azure connector. >> >> > > > >> > >> >> > > > >> > * Fixes and improvements post upgrade to AWS V2 SDK in >> S3AConnector. >> >> > > > >> > >> >> > > > >> > * This release includes Arm64 binaries. Please can anyone >> with >> >> > > > >> > >> >> > > > >> > compatible systems validate these. >> >> > > > >> > >> >> > > > >> > >> >> > > > >> > Note, because the arm64 binaries are built separately on a >> different >> >> > > > >> > >> >> > > > >> > platform and JVM, their jar files may not match those of >> the x86 >> >> > > > >> > >> >> > > > >> > release -and therefore the maven artifacts. I don't think >> this is >> >> > > > >> > >> >> > > > >> > an issue (the ASF actually releases source tarballs, the >> binaries >> >> > > are >> >> > > > >> > >> >> > > > >> > there for help only, though with the maven repo that's a bit >> >> > > blurred). >> >> > > > >> > >> >> > > > >> > >> >> > > > >> > The only way to be consistent would actually untar the >> x86.tar.gz, >> >> > > > >> > >> >> > > > >> > overwrite its binaries with the arm stuff, retar, sign and >> push out >> >> > > > >> > >> >> > > > >> > for the vote. Even automating that would be risky. >> >> > > > >> > >> >> > > > >> > >> >> > > > >> > Please try the release and vote. The vote will run for 5 >> days. >> >> > > > >> > >> >> > > > >> >> >> > > > > >> >> > > >> >> >> >> --------------------------------------------------------------------- >> >> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org >> >> For additional commands, e-mail: common-dev-h...@hadoop.apache.org >> >> >> >
