Thanx Mukund for sharing the details, I might have missed that discussion. Do mention the purpose on the website once we upload the lean jar, so that relevant people can use it.
I am holding my vote due to [3], that in general looks blocker to me, If I haven't messed up. Would be great if someone can help validate that I believe is the cause for [2] * Built from source * Verified checksums * Verified signatures * Validated no code diff b/w the git tag & src tar * Verified the output of `hadoop version` [1] * Checked the NOTICE & LICENSE file * Ran some basic HDFS shell commands * Ran some Erasure Coding related commands * Ran example jobs (TeraGen, TeraSort, WordCount) [2] * Browsed through the UI (NN, DN, RM, NM & JHS) * Skimmed over the contents of maven artifacts, Release Notes & ChangeLog * Discrepancy in contents of aarch64 tar & x86 tar [3] [1] The hadoop version output inside the aarch64 tar shows the source repository as Unknown, which isn't the case in other tar nor was the case in the last 3.4.0 release. Not blocking, but we should figure out & fix it in the next release. ``` ayushsaxena@ayushsaxena hadoop-3.4.1 % bin/hadoop version Hadoop 3.4.1 Source code repository Unknown -r Unknown Compiled by mthakur on 2024-10-10T08:24Z Compiled on platform linux-aarch_64 Compiled with protoc 3.23.4 >From source with checksum 7292fe9dba5e2e44e3a9f763fce3e680 ``` [2] While trying to start ResourceManager via the ARM binary, it is failing for me: ``` ayushsaxena@ayushsaxena hadoop-3.4.1 % bin/yarn resourcemanager Error: Could not find or load main class org.apache.hadoop.yarn.server.resourcemanager.ResourceManager ``` It didn't used to happen earlier, but I sorted it out by "export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:/Users/ayushsaxena/code/RC/hadoop-3.4.1/hadoop-3.4.1/share/hadoop/tools/lib/*", Considering I am not pro with YARN, I might have messed up the setup. My mapred settings have Tez related stuff as well, but the same works if I use the x86 binary.... [3] Investigating [2], I think the YARN Jars are missing in the aarch64 share/hadoop/yarn & they are there in the share/hadoop/tools directory in x86 ``` ayushsaxena@ayushsaxena hadoop-3.4.1 % ls x86/hadoop-3.4.1/share/hadoop/yarn csi hadoop-yarn-server-router-3.4.1.jar hadoop-yarn-api-3.4.1.jar hadoop-yarn-server-sharedcachemanager-3.4.1.jar hadoop-yarn-applications-catalog-webapp-3.4.1.war hadoop-yarn-server-tests-3.4.1.jar hadoop-yarn-applications-distributedshell-3.4.1.jar hadoop-yarn-server-timeline-pluginstorage-3.4.1.jar hadoop-yarn-applications-mawo-core-3.4.1.jar hadoop-yarn-server-web-proxy-3.4.1.jar hadoop-yarn-applications-unmanaged-am-launcher-3.4.1.jar hadoop-yarn-services-api-3.4.1.jar hadoop-yarn-client-3.4.1.jar hadoop-yarn-services-core-3.4.1.jar hadoop-yarn-common-3.4.1.jar lib hadoop-yarn-registry-3.4.1.jar sources hadoop-yarn-server-applicationhistoryservice-3.4.1.jar test hadoop-yarn-server-common-3.4.1.jar timelineservice hadoop-yarn-server-globalpolicygenerator-3.4.1.jar webapps hadoop-yarn-server-nodemanager-3.4.1.jar yarn-service-examples hadoop-yarn-server-resourcemanager-3.4.1.jar ayushsaxena@ayushsaxena hadoop-3.4.1 % ``` And in aarch64 they aren't there ``` ayushsaxena@ayushsaxena hadoop-3.4.1 % ls -l aarch/hadoop-3.4.1/share/hadoop/yarn total 0 drwxr-xr-x 3 ayushsaxena staff 96 Oct 10 15:34 csi drwxr-xr-x 47 ayushsaxena staff 1504 Oct 10 15:34 lib drwxr-xr-x 28 ayushsaxena staff 896 Oct 10 15:34 sources drwxr-xr-x 3 ayushsaxena staff 96 Oct 10 15:34 timelineservice drwxr-xr-x 3 ayushsaxena staff 96 Oct 10 15:34 webapps drwxr-xr-x 6 ayushsaxena staff 192 Oct 10 15:34 yarn-service-examples ayushsaxena@ayushsaxena hadoop-3.4.1 % ``` Can someone help double check this once? I downloaded the 3.4.0 binary & this is not the case over there. I did a build locally on aarch64 and there also the jars are present in the share/hadoop/yarn dir -Ayush On Fri, 18 Oct 2024 at 11:57, Mukund Madhav Thakur <mtha...@cloudera.com> wrote: > > Hi Ayush, > "lean" tar is a small tar file which doesn't contain the AWS SDK because the > size of AWS SDK is itself 500 MB. > This can ease usage for non AWS users. Even AWS users can add this jar > explicitly if desired. > > This is created using https://github.com/apache/hadoop-release-support which > is the new and easy way to create and validate hadoop releases > created by Steve during 3.4.0. Also the link to this is present in [1] . > There was a discussion around this during the RC2 testing. > > I will check with Steve on why we don't have the equivalent for aarch64. > > [1] https://cwiki.apache.org/confluence/display/HADOOP2/HowToRelease > > On Thu, Oct 17, 2024 at 10:00 PM Ayush Saxena <ayush...@gmail.com> wrote: >> >> What exactly is this "lean tar"? I couldn’t find any mention of it in >> [1], nor did I come across any thread establishing consensus on adding >> it, as we did for the aarch64 tar. It’s also not clear to me from the >> create-release script how it is getting generated. So, where is this >> coming from? Additionally, why don’t we have an equivalent for >> aarch64? >> >> -Ayush >> >> [1] https://cwiki.apache.org/confluence/display/HADOOP2/HowToRelease >> >> On Thu, 17 Oct 2024 at 13:21, Mukund Madhav Thakur >> <mtha...@cloudera.com.invalid> wrote: >> > >> > Adding my +1 (binding) after all the testing. >> > Ran a lot of validation workflows from hadoop-release-support all went >> > fine. >> > Verified checksum and signatures. >> > Build orc and reran the failing vectored io tests and it succeeded. >> > Build parquet and run the hadoop parquet module tests. >> > @Steve Loughran <ste...@apache.org> Build gcs and ran all the tests which >> > went fine. >> > >> > Also Ahmar is a committer so his vote can be counted as +1 as well. >> > >> > Thanks everyone for testing! Need just one more binding vote for this to be >> > released. >> > >> > >> > >> > >> > >> > On Thu, Oct 17, 2024 at 10:01 AM Sneha Vijayarajan < >> > sneha.vijayara...@gmail.com> wrote: >> > >> > > +1 (non-binding) >> > > >> > > Ran the ABFS driver tests and all the tests passed. >> > > >> > > ============================================================ >> > > HNS-OAuth >> > > ============================================================ >> > > >> > > [WARNING] Tests run: 154, Failures: 0, Errors: 0, Skipped: 2 >> > > [WARNING] Tests run: 646, Failures: 0, Errors: 0, Skipped: 82 >> > > [WARNING] Tests run: 433, Failures: 0, Errors: 0, Skipped: 57 >> > > >> > > ============================================================ >> > > HNS-SharedKey >> > > ============================================================ >> > > >> > > [WARNING] Tests run: 154, Failures: 0, Errors: 0, Skipped: 3 >> > > [WARNING] Tests run: 646, Failures: 0, Errors: 0, Skipped: 34 >> > > [WARNING] Tests run: 433, Failures: 0, Errors: 0, Skipped: 44 >> > > >> > > ============================================================ >> > > NonHNS-SharedKey >> > > ============================================================ >> > > >> > > [WARNING] Tests run: 154, Failures: 0, Errors: 0, Skipped: 9 >> > > [WARNING] Tests run: 630, Failures: 0, Errors: 0, Skipped: 274 >> > > [WARNING] Tests run: 433, Failures: 0, Errors: 0, Skipped: 47 >> > > >> > > ============================================================ >> > > AppendBlob-HNS-OAuth >> > > ============================================================ >> > > >> > > [WARNING] Tests run: 154, Failures: 0, Errors: 0, Skipped: 2 >> > > [WARNING] Tests run: 646, Failures: 0, Errors: 0, Skipped: 84 >> > > [WARNING] Tests run: 433, Failures: 0, Errors: 0, Skipped: 81 >> > > >> > > Thanks, >> > > Sneha Vijayarajan >> > > >> > > On Thu, Oct 17, 2024 at 8:57 AM Xiaoqiao He <hexiaoq...@apache.org> >> > > wrote: >> > > > >> > > > Update my vote to +1(binding). >> > > > >> > > > Addendum: Verified signature was correct for both src/binary/site >> > > tarball. >> > > > >> > > > >> > > > On Mon, Oct 14, 2024 at 5:26 PM Xiaoqiao He <hexiaoq...@apache.org> >> > > wrote: >> > > > >> > > > > Thanks Mukund and Steve for driving this release. >> > > > > >> > > > > +0. Will +1 when signature check passed. >> > > > > >> > > > > [Y] LICENSE files exist and NOTICE is included. >> > > > > [Y] Rat check is ok. mvn clean apache-rat:check >> > > > > [Y] Build the source code on Ubuntu and OpenJDK 11 by `mvn clean >> > > package >> > > > > -DskipTests -Pnative -Pdist -Dtar`. >> > > > > [Y] Setup pseudo cluster with HDFS and YARN. >> > > > > [Y] Run simple FsShell - mkdir/put/get/mv/rm and check the result. >> > > > > [Y] Run example mr jobs and check the result - Pi & wordcount. >> > > > > [Y] Spot-check and run some unit tests. >> > > > > [Y] Skimmed the Web UI of >> > > > > NameNode/DataNode/Resourcemanager/NodeManager >> > > > > etc. >> > > > > [Y] Skimmed over the contents of site documentation. >> > > > > [Y] Skimmed over the contents of maven repo. >> > > > > [N] Verified checksum passed but signature didn't pass. Did you use >> > > > > another key to sign? >> > > > > >> > > > >> hexiaoqiao@hexiaoqiao-ubuntu:~/Public/hadoop-3.4.1$ gpg --list-key | >> > > > >> grep -B2 -A1 "Mukund" >> > > > >> pub rsa4096 2023-09-28 [SC] [expires: 2027-09-28] >> > > > >> 41C7034C031FB3AF4BBFB5B89F070EBA4202CEC1 >> > > > >> uid [ unknown] Mukund Thakur <mtha...@apache.org> >> > > > >> sub rsa4096 2023-09-28 [E] [expires: 2027-09-28] >> > > > >> hexiaoqiao@hexiaoqiao-ubuntu:~/Public/hadoop-3.4.1$ gpg --verify >> > > > >> hadoop-3.4.1.tar.gz.asc hadoop-3.4.1.tar.gz >> > > > >> gpg: Signature made 2024年10月10日 星期四 01时10分30秒 CST >> > > > >> gpg: using RSA key >> > > 53931DAA708291409958BD474D22BB7D32882201 >> > > > >> gpg: Can't check signature: No public key >> > > > > >> > > > > >> > > > > Best Regards, >> > > > > - He Xiaoqiao >> > > > > >> > > > > On Fri, Oct 11, 2024 at 1:31 AM Steve Loughran >> > > <ste...@cloudera.com.invalid> >> > > > > wrote: >> > > > > >> > > > >> Hney, you did the -lean one too! nice >> > > > >> >> > > > >> Anyway, yes I'll test. I will try not to find new problems >> > > > >> >> > > > >> >> > > > >> On Thu, 10 Oct 2024 at 12:18, Mukund Madhav Thakur >> > > > >> <mtha...@cloudera.com.invalid> wrote: >> > > > >> >> > > > >> > Apache Hadoop 3.4.1 >> > > > >> > >> > > > >> > >> > > > >> > With help of Steve, I have put together a release candidate (RC3) >> > > for >> > > > >> > Hadoop 3.4.1. >> > > > >> > >> > > > >> > >> > > > >> > What we would like is for anyone who can to verify the tarballs, >> > > > >> especially >> > > > >> > >> > > > >> > anyone who can try the arm64 binaries as we want to include them >> > > too. >> > > > >> > >> > > > >> > >> > > > >> > The RC is available at: >> > > > >> > >> > > > >> > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.1-RC3/ >> > > > >> > >> > > > >> > >> > > > >> > The git tag is release-3.4.1-RC3, commit >> > > > >> > 4d7825309348956336b8f06a08322b78422849b1 >> > > > >> > >> > > > >> > >> > > > >> > The maven artifacts are staged at >> > > > >> > >> > > > >> > >> > > https://repository.apache.org/content/repositories/orgapachehadoop-1430 >> > > > >> > >> > > > >> > >> > > > >> > You can find my public key at: >> > > > >> > >> > > > >> > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS >> > > > >> > >> > > > >> > >> > > > >> > Change log >> > > > >> > >> > > > >> > >> > > > >> >> > > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.1-RC3/CHANGELOG.md >> > > > >> > >> > > > >> > >> > > > >> > Release notes >> > > > >> > >> > > > >> > >> > > > >> > >> > > > >> >> > > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.1-RC3/RELEASENOTES.md >> > > > >> > >> > > > >> > >> > > > >> > This is off branch-3.4.1 >> > > > >> > >> > > > >> > >> > > > >> > Key changes include >> > > > >> > >> > > > >> > >> > > > >> > * Bulk Delete API. >> > > https://issues.apache.org/jira/browse/HADOOP-18679 >> > > > >> > >> > > > >> > * Fixes and enhancements in Vectored IO API. >> > > > >> > >> > > > >> > * Improvements in Hadoop Azure connector. >> > > > >> > >> > > > >> > * Fixes and improvements post upgrade to AWS V2 SDK in >> > > > >> > S3AConnector. >> > > > >> > >> > > > >> > * This release includes Arm64 binaries. Please can anyone with >> > > > >> > >> > > > >> > compatible systems validate these. >> > > > >> > >> > > > >> > >> > > > >> > Note, because the arm64 binaries are built separately on a >> > > > >> > different >> > > > >> > >> > > > >> > platform and JVM, their jar files may not match those of the x86 >> > > > >> > >> > > > >> > release -and therefore the maven artifacts. I don't think this is >> > > > >> > >> > > > >> > an issue (the ASF actually releases source tarballs, the binaries >> > > are >> > > > >> > >> > > > >> > there for help only, though with the maven repo that's a bit >> > > blurred). >> > > > >> > >> > > > >> > >> > > > >> > The only way to be consistent would actually untar the x86.tar.gz, >> > > > >> > >> > > > >> > overwrite its binaries with the arm stuff, retar, sign and push >> > > > >> > out >> > > > >> > >> > > > >> > for the vote. Even automating that would be risky. >> > > > >> > >> > > > >> > >> > > > >> > Please try the release and vote. The vote will run for 5 days. >> > > > >> > >> > > > >> >> > > > > >> > > >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org >> For additional commands, e-mail: common-dev-h...@hadoop.apache.org >> --------------------------------------------------------------------- To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
