Apache Hadoop qbt Report: branch-2.10+JDK7 on Linux/x86_64
For more details, see https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/665/ No changes -1 overall The following subsystems voted -1: asflicense hadolint mvnsite pathlen unit The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck whitespace The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: Failed junit tests : hadoop.fs.TestTrash hadoop.fs.TestFileUtil hadoop.hdfs.server.datanode.TestDirectoryScanner hadoop.hdfs.qjournal.server.TestJournalNodeRespectsBindHostKeys hadoop.hdfs.TestDecommission hadoop.hdfs.TestClose hadoop.cli.TestHDFSCLI hadoop.hdfs.server.blockmanagement.TestReplicationPolicyWithUpgradeDomain hadoop.hdfs.TestRead hadoop.hdfs.server.namenode.ha.TestEditLogTailer hadoop.hdfs.TestBlockStoragePolicy hadoop.hdfs.TestDFSClientRetries hadoop.hdfs.TestExternalBlockReader hadoop.metrics2.sink.TestRollingFileSystemSinkWithHdfs hadoop.hdfs.client.impl.TestBlockReaderLocal hadoop.hdfs.TestBlockTokenWrappingQOP hadoop.fs.contract.router.TestRouterHDFSContractSetTimes hadoop.mapreduce.jobhistory.TestHistoryViewerPrinter hadoop.mapreduce.lib.input.TestLineRecordReader hadoop.mapreduce.v2.hs.TestJobHistory hadoop.mapreduce.v2.hs.webapp.TestHsWebServicesAttempts hadoop.tools.TestDistCpSystem hadoop.resourceestimator.solver.impl.TestLpSolver hadoop.resourceestimator.service.TestResourceEstimatorService hadoop.yarn.sls.TestSLSRunner hadoop.yarn.server.resourcemanager.TestClientRMService hadoop.yarn.server.resourcemanager.monitor.invariants.TestMetricsInvariantChecker cc: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/665/artifact/out/diff-compile-cc-root.txt [4.0K] javac: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/665/artifact/out/diff-compile-javac-root.txt [476K] checkstyle: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/665/artifact/out/diff-checkstyle-root.txt [14M] hadolint: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/665/artifact/out/diff-patch-hadolint.txt [4.0K] mvnsite: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/665/artifact/out/patch-mvnsite-root.txt [556K] pathlen: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/665/artifact/out/pathlen.txt [12K] pylint: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/665/artifact/out/diff-patch-pylint.txt [20K] shellcheck: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/665/artifact/out/diff-patch-shellcheck.txt [72K] whitespace: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/665/artifact/out/whitespace-eol.txt [12M] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/665/artifact/out/whitespace-tabs.txt [1.3M] javadoc: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/665/artifact/out/patch-javadoc-root.txt [40K] unit: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/665/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt [220K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/665/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt [560K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/665/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt [20K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/665/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs_src_contrib_bkjournal.txt [0] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/665/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app.txt [0] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/665/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt [104K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/665/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-hs.txt [128K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/665/artifact/out/patch-unit-hadoop-tools_hadoop-azure.txt [20K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/665/artifact/out/patch-un
[ANNOUNCE] Apache Hadoop 3.3.3 release
On behalf of the Apache Hadoop Project Management Committee, I'm pleased to announce that Hadoop 3.3.3 has been released: https://hadoop.apache.org/release/3.3.3.html This is the third stable release of the Apache Hadoop 3.3 line. It contains 23 bug fixes, improvements and enhancements since 3.3.2. This is primarily a security update; for this reason, upgrading is strongly advised. Users are encouraged to read the overview of major changes[1] since 3.3.2. For details of bug fixes, improvements, and other enhancements since the previous 3.3.2 release, please check release notes[2] and changelog[3]. [1]: /docs/r3.3.3/index.html [2]: http://hadoop.apache.org/docs/r3.3.3/hadoop-project-dist/hadoop-common/release/3.3.3/RELEASENOTES.3.3.3.html [3]: http://hadoop.apache.org/docs/r3.3.3/hadoop-project-dist/hadoop-common/release/3.3.3/CHANGELOG.3.3.3.html As the release notes highlight, this release contains HADOOP-18088 "Replace log4j 1.x with reload4j" https://issues.apache.org/jira/browse/HADOOP-18088 This ensures that the version of log4j shipped is free of known CVEs. the standard log4j 1.2.17 has some known CVEs in classes which were never uses; reload4j cuts them out. Audit scanning tools should stop highlighting perceived risks here. If you are using maven exclusions to manage logging libraries, or were otherwise replacing the log4j artifacts in deployments, note the different library/artifact names which need to be handled. Many thanks to everyone who helped in this release by supplying patches, reviewing them, helping get this release building and testing reviewing the final artifacts. Steve
Re: [DISCUSS] Enabling all platform builds in CI for all Hadoop PRs
Have raised HADOOP-18241 There is a WIP PR linked to it as well https://issues.apache.org/jira/browse/HADOOP-18241 On Wed, 11 May 2022 at 01:28, Ayush Saxena wrote: > +1 for moving to Java 11 for trunk. > If nobody objects by EOW will create a jira for that > > -Ayush > > On 10-May-2022, at 11:18 PM, Gautham Banasandra > wrote: > > > +1 for moving to Java 11. > > Thanks, > --Gautham > > On Mon, 9 May 2022 at 19:50, Steve Vaughan wrote: > >> +1 on Java 11 for branch-3.3 >> >> I'd also like to start addressing the test instability by addressing >> collisions between tests. Currently the tests are working in the same >> shared resources either directly through the system properties like >> test.build.data, indirectly through utility classes like GenericTestUtils, >> or modifying shared resources within target/test-classes. I've been >> working towards cleaning up flaky tests, and I plan on submitting some >> patches today. >> -- >> *From:* Steve Loughran >> *Sent:* Monday, May 9, 2022 7:50 AM >> *To:* Ayush Saxena >> *Cc:* Gautham Banasandra ; Hadoop Common < >> common-...@hadoop.apache.org>; Hdfs-dev ; >> yarn-dev ; d...@hadoop.apache.org < >> d...@hadoop.apache.org> >> *Subject:* Re: [DISCUSS] Enabling all platform builds in CI for all >> Hadoop PRs >> >> how about for trunk we -wait for it- declare it java11 only? >> >> do that and we cut out a lot of the CI build. >> >> we would have to mandate a test run through branch-3.3 before any >> cherrypick there >> >> On Sat, 7 May 2022 at 11:19, Ayush Saxena wrote: >> >> > Three for trunk: >> > Java-8 >> > >> > https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/ >> > >> > Java-11 >> > >> > https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java11-linux-x86_64/ >> > >> > ARM >> > >> > >> https://ci-hadoop.apache.org/view/Hadoop/job/hadoop-qbt-linux-ARM-trunk/ >> > >> > -Ayush >> > >> > > >> > > On 07-May-2022, at 11:49 AM, Gautham Banasandra >> > wrote: >> > > >> > > Hi all, >> > > >> > > Although validating cross platform builds at pre-commit >> > > would be the most effective approach, I understand the >> > > huge disadvantage caused by the slowdown. The best >> > > way to tackle this would be to enable parallel builds for >> > > the different platforms. I had given it a shot about a year >> > > ago[1], it didn't go well and ran into all sorts of random >> > > errors. I think we should make the parallel builds run on >> > > different agents as opposed to starting the builds parallelly >> > > on the same agent (which is what I had done earlier). >> > > >> > > So, I'll settle down to integrating the full suite of platform >> > > builds into the nightly builds. Could anyone please point >> > > me to the Jenkins job for this? >> > > >> > > [1] = https://github.com/apache/hadoop/pull/3166 >> > > >> > > Thanks, >> > > --Gautham >> > > >> > >> On Fri, 6 May 2022 at 21:04, Ayush Saxena >> wrote: >> > >> >> > >> From functional point of view it does makes sense to validate all the >> > >> platforms as part of the builds, but the Pre commits builds taking >> time >> > is >> > >> now no longer a small things, In past one year or may be two, we have >> > >> already increased it more than twice as compared to what it was >> before, >> > If >> > >> someone has a change in HDFS, which includes both hdfs-client & >> > >> hadoop-hdfs, it takes more than 5 hours, which long back was around 2 >> > hours. >> > >> With the current state, I don't think we should just go and add these >> > >> extra overheads. Having them as part of the nightly builds does makes >> > sense >> > >> for now. >> > >> >> > >> In future if we feel there is a strong need for this and we start to >> see >> > >> very frequent failures in some other platforms and we are left with >> no >> > >> other option but to integrate it in our pre-commit jobs, we can >> explore >> > >> having these build phases running in parallel, along with trying >> other >> > >> phases also to run in parallel like compilation/javadoc builds of >> JDK-8 >> > & >> > >> JDK-11 can run in parallel and may be explore other opportunities as >> > well >> > >> to compensate for this time. >> > >> >> > >> For now lets just integrate it our nightly builds only and circle >> back >> > >> again here in future if the need be. >> > >> >> > >> -Ayush >> > >> >> > >>> On Fri, 6 May 2022 at 20:44, Wei-Chiu Chuang >> > wrote: >> > >>> >> > >>> Running builds for all platforms for each and every PR seems too >> > >>> excessive. >> > >>> >> > >>> How about doing all platform builds in the nightly jobs? >> > >>> >> > >>> On Fri, May 6, 2022 at 8:02 AM Steve Loughran >> > > > >> > >>> wrote: >> > >>> >> > I'm not enthusiastic here as it not only makes the builds slower, >> it >> > reduces the #of builds we can through a day >> > >> > one thing I am wondering is could we remove java8 support on some >> > >>> branches? >> > >> > make branch 3.3.2.x (i.e the
