Re: [VOTE] Release Apache Hadoop 3.3.1 RC3
I found that the top page of site documentation (index.html on the top directory) has the same contents of 3.0.0 (https://hadoop.apache.org/docs/r3.0.0/). While I think it is not worth for sinking the RC3, you can update the page based on the one for 3.3.0 (http://hadoop.apache.org/docs/r3.3.0/) if you need to cut RC4. Thanks, Masatake Iwasaki On 2021/06/01 19:29, Wei-Chiu Chuang wrote: Hi community, This is the release candidate RC3 of Apache Hadoop 3.3.1 line. All blocker issues have been resolved [1] again. There are 2 additional issues resolved for RC3: * Revert "MAPREDUCE-7303. Fix TestJobResourceUploader failures after HADOOP-16878 * Revert "HADOOP-16878. FileUtil.copy() to throw IOException if the source and destination are the same There are 4 issues resolved for RC2: * HADOOP-17666. Update LICENSE for 3.3.1 * MAPREDUCE-7348. TestFrameworkUploader#testNativeIO fails. (#3053) * Revert "HADOOP-17563. Update Bouncy Castle to 1.68. (#2740)" (#3055) * HADOOP-17739. Use hadoop-thirdparty 1.1.1. (#3064) The Hadoop-thirdparty 1.1.1, as previously mentioned, contains two extra fixes compared to hadoop-thirdparty 1.1.0: * HADOOP-17707. Remove jaeger document from site index. * HADOOP-17730. Add back error_prone *RC tag is release-3.3.1-RC3 https://github.com/apache/hadoop/releases/tag/release-3.3.1-RC3 *The RC3 artifacts are at*: https://home.apache.org/~weichiu/hadoop-3.3.1-RC3/ ARM artifacts: https://home.apache.org/~weichiu/hadoop-3.3.1-RC3-arm/ *The maven artifacts are hosted here:* https://repository.apache.org/content/repositories/orgapachehadoop-1320/ *My public key is available here:* https://dist.apache.org/repos/dist/release/hadoop/common/KEYS Things I've verified: * all blocker issues targeting 3.3.1 have been resolved. * stable/evolving API changes between 3.3.0 and 3.3.1 are compatible. * LICENSE and NOTICE files checked * RELEASENOTES and CHANGELOG * rat check passed. * Built HBase master branch on top of Hadoop 3.3.1 RC2, ran unit tests. * Built Ozone master on top fo Hadoop 3.3.1 RC2, ran unit tests. * Extra: built 50 other open source projects on top of Hadoop 3.3.1 RC2. Had to patch some of them due to commons-lang migration (Hadoop 3.2.0) and dependency divergence. Issues are being identified but so far nothing blocker for Hadoop itself. Please try the release and vote. The vote will run for 5 days. My +1 to start, [1] https://issues.apache.org/jira/issues/?filter=12350491 [2] https://github.com/apache/hadoop/compare/release-3.3.1-RC1...release-3.3.1-RC3 - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 3.3.1 RC3
For someone having similar issue. When I tested the RC3, ResourceManager failed to start due to following exception.:: 2021-06-07 10:50:54,715 ERROR org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error starting ResourceManager java.lang.IllegalAccessError: tried to access class org.bouncycastle.asn1.DEROutputStream from class org.bouncycastle.cert.CertUtils at org.bouncycastle.cert.CertUtils.generateSig(Unknown Source) at org.bouncycastle.cert.CertUtils.generateFullCert(Unknown Source) at org.bouncycastle.cert.X509v3CertificateBuilder.build(Unknown Source) at org.apache.hadoop.yarn.server.webproxy.ProxyCA.createCert(ProxyCA.java:180) ... The cause turned out to be my local environment. When I was working on HADOOP-17609, in order to try SM4 feature, I installed bcprov-ext-jdk15on-168.jar to $JAVA_HOME/jre/lib/ext and added the line "security.provider.10=org.bouncycastle.jce.provider.BouncyCastleProvider" to $JAVA_HOME/jre/lib/security/java.security based on the comment on HDFS-15098 [1] (and fotgot doing so). Removing the bcprov-ext-jdk15on-168.jar from $JAVA_HOME/jre/lib/ext fixed the issue. [1] https://issues.apache.org/jira/browse/HDFS-15098?focusedCommentId=17112893&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17112893 Masatake Iwasaki On 2021/06/07 18:04, Masatake Iwasaki wrote: I found that the top page of site documentation (index.html on the top directory) has the same contents of 3.0.0 (https://hadoop.apache.org/docs/r3.0.0/). While I think it is not worth for sinking the RC3, you can update the page based on the one for 3.3.0 (http://hadoop.apache.org/docs/r3.3.0/) if you need to cut RC4. Thanks, Masatake Iwasaki On 2021/06/01 19:29, Wei-Chiu Chuang wrote: Hi community, This is the release candidate RC3 of Apache Hadoop 3.3.1 line. All blocker issues have been resolved [1] again. There are 2 additional issues resolved for RC3: * Revert "MAPREDUCE-7303. Fix TestJobResourceUploader failures after HADOOP-16878 * Revert "HADOOP-16878. FileUtil.copy() to throw IOException if the source and destination are the same There are 4 issues resolved for RC2: * HADOOP-17666. Update LICENSE for 3.3.1 * MAPREDUCE-7348. TestFrameworkUploader#testNativeIO fails. (#3053) * Revert "HADOOP-17563. Update Bouncy Castle to 1.68. (#2740)" (#3055) * HADOOP-17739. Use hadoop-thirdparty 1.1.1. (#3064) The Hadoop-thirdparty 1.1.1, as previously mentioned, contains two extra fixes compared to hadoop-thirdparty 1.1.0: * HADOOP-17707. Remove jaeger document from site index. * HADOOP-17730. Add back error_prone *RC tag is release-3.3.1-RC3 https://github.com/apache/hadoop/releases/tag/release-3.3.1-RC3 *The RC3 artifacts are at*: https://home.apache.org/~weichiu/hadoop-3.3.1-RC3/ ARM artifacts: https://home.apache.org/~weichiu/hadoop-3.3.1-RC3-arm/ *The maven artifacts are hosted here:* https://repository.apache.org/content/repositories/orgapachehadoop-1320/ *My public key is available here:* https://dist.apache.org/repos/dist/release/hadoop/common/KEYS Things I've verified: * all blocker issues targeting 3.3.1 have been resolved. * stable/evolving API changes between 3.3.0 and 3.3.1 are compatible. * LICENSE and NOTICE files checked * RELEASENOTES and CHANGELOG * rat check passed. * Built HBase master branch on top of Hadoop 3.3.1 RC2, ran unit tests. * Built Ozone master on top fo Hadoop 3.3.1 RC2, ran unit tests. * Extra: built 50 other open source projects on top of Hadoop 3.3.1 RC2. Had to patch some of them due to commons-lang migration (Hadoop 3.2.0) and dependency divergence. Issues are being identified but so far nothing blocker for Hadoop itself. Please try the release and vote. The vote will run for 5 days. My +1 to start, [1] https://issues.apache.org/jira/issues/?filter=12350491 [2] https://github.com/apache/hadoop/compare/release-3.3.1-RC1...release-3.3.1-RC3 - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
Re: [VOTE] Hadoop 3.1.x EOL
+1 (binding) -Eric On Thursday, June 3, 2021, 1:14:51 AM CDT, Akira Ajisaka wrote: Dear Hadoop developers, Given the feedback from the discussion thread [1], I'd like to start an official vote thread for the community to vote and start the 3.1 EOL process. What this entails: (1) an official announcement that no further regular Hadoop 3.1.x releases will be made after 3.1.4. (2) resolve JIRAs that specifically target 3.1.5 as won't fix. This vote will run for 7 days and conclude by June 10th, 16:00 JST [2]. Committers are eligible to cast binding votes. Non-committers are welcomed to cast non-binding votes. Here is my vote, +1 [1] https://s.apache.org/w9ilb [2] https://www.timeanddate.com/worldclock/fixedtime.html?msg=4&iso=20210610T16&p1=248 Regards, Akira - To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org
Re: [E] [VOTE] Hadoop 3.1.x EOL
+1 (binding) On Thu, Jun 3, 2021 at 1:14 AM Akira Ajisaka wrote: > Dear Hadoop developers, > > Given the feedback from the discussion thread [1], I'd like to start > an official vote > thread for the community to vote and start the 3.1 EOL process. > > What this entails: > > (1) an official announcement that no further regular Hadoop 3.1.x releases > will be made after 3.1.4. > (2) resolve JIRAs that specifically target 3.1.5 as won't fix. > > This vote will run for 7 days and conclude by June 10th, 16:00 JST [2]. > > Committers are eligible to cast binding votes. Non-committers are welcomed > to cast non-binding votes. > > Here is my vote, +1 > > [1] > https://urldefense.proofpoint.com/v2/url?u=https-3A__s.apache.org_w9ilb&d=DwIBaQ&c=sWW_bEwW_mLyN3Kx2v57Q8e-CRbmiT9yOhqES_g_wVY&r=b6gUZYewojO-9YMJdyeI_g&m=XwkYgkbxoxl9GFqT_D4JQxurNTSSFm9Q2nNXBsx7kvQ&s=_t7jj-Mw4LwNSdTgrKrtA5aeB_8t2feBuG89IjMHRYE&e= > [2] > https://urldefense.proofpoint.com/v2/url?u=https-3A__www.timeanddate.com_worldclock_fixedtime.html-3Fmsg-3D4-26iso-3D20210610T16-26p1-3D248&d=DwIBaQ&c=sWW_bEwW_mLyN3Kx2v57Q8e-CRbmiT9yOhqES_g_wVY&r=b6gUZYewojO-9YMJdyeI_g&m=XwkYgkbxoxl9GFqT_D4JQxurNTSSFm9Q2nNXBsx7kvQ&s=Z7y-vP3_I6MdDApyRBz6c_tBnVtuvyjZB_k5q1zaR2c&e= > > Regards, > Akira > > - > To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org > For additional commands, e-mail: common-dev-h...@hadoop.apache.org > >
Re: [VOTE] Hadoop 3.1.x EOL
On Thu, 3 Jun 2021 at 07:14, Akira Ajisaka wrote: > Dear Hadoop developers, > > Given the feedback from the discussion thread [1], I'd like to start > an official vote > thread for the community to vote and start the 3.1 EOL process. > > What this entails: > > (1) an official announcement that no further regular Hadoop 3.1.x releases > will be made after 3.1.4. > (2) resolve JIRAs that specifically target 3.1.5 as won't fix. > > This vote will run for 7 days and conclude by June 10th, 16:00 JST [2]. > > Committers are eligible to cast binding votes. Non-committers are welcomed > to cast non-binding votes. > > Here is my vote, +1 > +1 (binding) > >
Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86_64
For more details, see https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/531/ No changes -1 overall The following subsystems voted -1: blanks pathlen unit xml The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: XML : Parsing Error(s): hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-excerpt.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags2.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-sample-output.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/fair-scheduler-invalid.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/yarn-site-with-invalid-allocation-file-ref.xml Failed CTEST tests : test_libhdfs_threaded_hdfspp_test_shim_static test_hdfspp_mini_dfs_smoke_hdfspp_test_shim_static libhdfs_mini_stress_valgrind_hdfspp_test_static memcheck_libhdfs_mini_stress_valgrind_hdfspp_test_static test_libhdfs_mini_stress_hdfspp_test_shim_static test_hdfs_ext_hdfspp_test_shim_static x_platform_utils_test x_platform_syscall_test Failed junit tests : hadoop.metrics2.source.TestJvmMetrics hadoop.hdfs.server.namenode.TestBlockPlacementPolicyRackFaultTolerant hadoop.yarn.server.timelineservice.storage.common.TestHBaseTimelineStorageUtils hadoop.yarn.server.router.clientrm.TestFederationClientInterceptor hadoop.yarn.csi.client.TestCsiClient hadoop.tools.dynamometer.workloadgenerator.TestWorkloadGenerator hadoop.tools.dynamometer.TestDynamometerInfra hadoop.tools.dynamometer.blockgenerator.TestBlockGen hadoop.tools.dynamometer.TestDynamometerInfra hadoop.tools.dynamometer.blockgenerator.TestBlockGen hadoop.tools.dynamometer.workloadgenerator.TestWorkloadGenerator cc: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/531/artifact/out/results-compile-cc-root.txt [96K] javac: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/531/artifact/out/results-compile-javac-root.txt [380K] blanks: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/531/artifact/out/blanks-eol.txt [13M] https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/531/artifact/out/blanks-tabs.txt [2.0M] checkstyle: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/531/artifact/out/results-checkstyle-root.txt [16M] pathlen: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/531/artifact/out/results-pathlen.txt [16K] pylint: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/531/artifact/out/results-pylint.txt [20K] shellcheck: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/531/artifact/out/results-shellcheck.txt [28K] xml: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/531/artifact/out/xml.txt [24K] javadoc: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/531/artifact/out/results-javadoc-javadoc-root.txt [408K] unit: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/531/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt [256K] https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/531/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt [540K] https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/531/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-native-client.txt [216K] https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/531/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice-hbase_hadoop-yarn-server-timelineservice-hbase-client.txt [24K] https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/531/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-router.txt [24K] https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/531/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-csi.txt [20K] https://ci-hadoop.apache.org/job/hadoop-qbt-tr
Apache Hadoop qbt Report: trunk+JDK11 on Linux/x86_64
For more details, see https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java11-linux-x86_64/181/ [Jun 5, 2021 10:05:29 PM] (Akira Ajisaka) Fix container-executor [Jun 7, 2021 2:51:29 AM] (noreply) MAPREDUCE-7350. Replace Guava Lists usage by Hadoop's own Lists in hadoop-mapreduce-project (#3074) [Jun 7, 2021 4:24:09 AM] (noreply) HADOOP-17743. Replace Guava Lists usage by Hadoop's own Lists in hadoop-common, hadoop-tools and cloud-storage projects (#3072) -1 overall The following subsystems voted -1: blanks mvnsite pathlen spotbugs unit xml The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: XML : Parsing Error(s): hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-excerpt.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags2.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-sample-output.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/fair-scheduler-invalid.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/yarn-site-with-invalid-allocation-file-ref.xml spotbugs : module:hadoop-hdfs-project/hadoop-hdfs Redundant nullcheck of oldLock, which is known to be non-null in org.apache.hadoop.hdfs.server.datanode.DataStorage.isPreUpgradableLayout(Storage$StorageDirectory) Redundant null check at DataStorage.java:is known to be non-null in org.apache.hadoop.hdfs.server.datanode.DataStorage.isPreUpgradableLayout(Storage$StorageDirectory) Redundant null check at DataStorage.java:[line 694] org.apache.hadoop.hdfs.server.namenode.top.window.RollingWindowManager$UserCounts doesn't override java.util.ArrayList.equals(Object) At RollingWindowManager.java:At RollingWindowManager.java:[line 1] spotbugs : module:hadoop-yarn-project/hadoop-yarn Redundant nullcheck of it, which is known to be non-null in org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService.recoverTrackerResources(LocalResourcesTracker, NMStateStoreService$LocalResourceTrackerState) Redundant null check at ResourceLocalizationService.java:is known to be non-null in org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService.recoverTrackerResources(LocalResourcesTracker, NMStateStoreService$LocalResourceTrackerState) Redundant null check at ResourceLocalizationService.java:[line 343] Redundant nullcheck of it, which is known to be non-null in org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService.recoverTrackerResources(LocalResourcesTracker, NMStateStoreService$LocalResourceTrackerState) Redundant null check at ResourceLocalizationService.java:is known to be non-null in org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService.recoverTrackerResources(LocalResourcesTracker, NMStateStoreService$LocalResourceTrackerState) Redundant null check at ResourceLocalizationService.java:[line 356] Boxed value is unboxed and then immediately reboxed in org.apache.hadoop.yarn.server.timelineservice.storage.common.ColumnRWHelper.readResultsWithTimestamps(Result, byte[], byte[], KeyConverter, ValueConverter, boolean) At ColumnRWHelper.java:then immediately reboxed in org.apache.hadoop.yarn.server.timelineservice.storage.common.ColumnRWHelper.readResultsWithTimestamps(Result, byte[], byte[], KeyConverter, ValueConverter, boolean) At ColumnRWHelper.java:[line 333] spotbugs : module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server Redundant nullcheck of it, which is known to be non-null in org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService.recoverTrackerResources(LocalResourcesTracker, NMStateStoreService$LocalResourceTrackerState) Redundant null check at ResourceLocalizationService.java:is known to be non-null in org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService.recoverTrackerResources(LocalResourcesTracker, NMStateStoreService$LocalResourceTrackerState) Redundant null check at ResourceLocalizationService.java:[line 343] Redundant nullcheck of it, which is known to be non-null in org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLoca
