Ozone tests: A few issues were found during the test. (1) HADOOP-17730 <https://issues.apache.org/jira/browse/HADOOP-17730> as mentioned previously, this is a Hadoop side problem, which requires a new Hadoop thirdparty release. (2) class path conflicts. As Ozone was split from Hadoop, it copies a few test utility from Hadoop but didn't relocate the classpaths. This is an Ozone-side problem. (3) mockito2 migration caused by HADOOP-14178 <https://issues.apache.org/jira/browse/HADOOP-14178>. Again, Ozone's test code partially depends on Hadoop, which previously used Mockito 1. There's some migration work required at Ozone side.
HBase: 1. HBASE-25928 <https://issues.apache.org/jira/browse/HBASE-25928> behavior change in Configuration in Hadoop 3.3.0 caused by 2. HADOOP-15708 <https://issues.apache.org/jira/browse/HADOOP-15708>. I think we better document this somewhere, because Configuration class is used widely. Additionally, 1. MAPREDUCE-7348 <https://issues.apache.org/jira/browse/MAPREDUCE-7348> caused by commons-io update 2. 1. 1. HADOOP-17563 <https://issues.apache.org/jira/browse/HADOOP-17563> bumped bouncycastle version but it breaks Spark and HBase. 3. 1. HADOOP-17292 <https://issues.apache.org/jira/browse/HADOOP-17292> replaced lz4 codec with lz4-java. But it's declared as provided scope, requiring downstream applications such as HBase to declare the dependency explicitly. It would be best to add a release note. I propose to start a RC2 to include (1) hadoop-thirdparty 1.1.1 (release vote in progress), (2) MAPREDUCE-7348, (3) revert HADOOP-17563, (4) add release notes to HADOOP-17292 and HADOOP-14178. Please review the hadoop-thirdparty 1.1.1 RC0 and cast a vote. Thanks! On Wed, May 26, 2021 at 9:39 AM Wei-Chiu Chuang <weic...@apache.org> wrote: > > Found a few issues building downstreams > > Ozone: HADOOP-17730 <https://issues.apache.org/jira/browse/HADOOP-17730> -- > we may have to make an hadoop-thirdparty 1.1.1 to address this problem. > > HBase: HBASE-25908 <https://issues.apache.org/jira/browse/HBASE-25908> -- > this is mostly a HBase side of problem. > > hbase-filesystem (HBOSS) does not compile because of the changing API on > the Hadoop side. This is expected since the interface isn't stable. > > Tez -- does not compile due to TEZ-4298 > <https://issues.apache.org/jira/browse/TEZ-4298> > > === > HBase test failures: > > TestBackupSmallTests > java.lang.NoClassDefFoundError: > org/bouncycastle/asn1/edec/EdECObjectIdentifiers > > --> HBase needs to bump the runtime version of bouncycastle to 1.68 to > find the missing class file. > > TestCompressionTest > java.lang.AssertionError > at > org.apache.hadoop.hbase.util.TestCompressionTest.testTestCompression(TestCompressionTest.java:89) > (This is due to the lz4/lzo compression codec change) > The lz4-java is in provide scope. Downstream applications that use > lz4-java must declare lz4-java explicitly. I think we need to add a release > note for it. > > > TestHBaseConfiguration > java.lang.AssertionError: expected null, but was:<1000> > at > org.apache.hadoop.hbase.TestHBaseConfiguration.testDeprecatedConfigurations(TestHBaseConfiguration.java:150) > > On Mon, May 24, 2021 at 10:36 PM Wei-Chiu Chuang <weic...@apache.org> > wrote: > >> Hi community, >> >> This is the release candidate RC1 of Apache Hadoop 3.3.1 line. All >> blocker issues have been resolved. >> >> It contains 697 fixed jira issues [2] since 3.3.0 which include a lot of >> features and improvements(read the full set of release notes). >> >> Below feature additions are the highlights of the release. >> Using lz4-java in Lz4Codec >> <https://issues.apache.org/jira/browse/HADOOP-17292> >> Using snappy-java in SnappyCodec >> <https://issues.apache.org/jira/browse/HADOOP-17125> >> Provide Regex Based Mount Point In Inode Tree >> <https://issues.apache.org/jira/browse/HADOOP-15891> >> Add Public IOStatistics API >> <https://issues.apache.org/jira/browse/HADOOP-16830> >> ABFS: Delegation SAS Generator Updates >> <https://issues.apache.org/jira/browse/HADOOP-17076> >> ABFS: Delegation SAS generator for integration with Ranger >> <https://issues.apache.org/jira/browse/HADOOP-16916> >> Über-jira: S3A Hadoop 3.3.1 features >> <https://issues.apache.org/jira/browse/HADOOP-16829> >> Add Metrics to HttpFS Server >> <https://issues.apache.org/jira/browse/HDFS-15711> >> EC: Verify EC reconstruction correctness on DataNode >> <https://issues.apache.org/jira/browse/HDFS-15759> >> Standby NameNode process getBlocks request to reduce Active load >> <https://issues.apache.org/jira/browse/HDFS-13183> >> LocatedFileStatusFetcher to collect/publish IOStatistics >> <https://issues.apache.org/jira/browse/MAPREDUCE-7315> >> >> *RC tag is release-3.3.1-RC1 >> https://github.com/apache/hadoop/releases/tag/release-3.3.1-RC1 >> >> *The RC1 artifacts are at*: >> https://home.apache.org/~weichiu/hadoop-3.3.1-RC1/ >> ARM artifacts: https://home.apache.org/~weichiu/hadoop-3.3.1-RC1-arm/ >> >> >> *The maven artifacts are hosted here:* >> https://repository.apache.org/content/repositories/orgapachehadoop-1314/ >> >> *My public key is available here:* >> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS >> >> [1] https://issues.apache.org/jira/issues/?filter=12350491 >> [2] >> https://issues.apache.org/jira/issues/?jql=project%20in%20(YARN%2C%20HADOOP%2C%20MAPREDUCE%2C%20HDFS)%20AND%20fixVersion%20in%20( >> 3.3.1)%20AND%20status%20%3D%20Resolved%20ORDER%20BY%0AfixVersion%20ASC >> >> >> My ask: >> (1) please use the bits to test downstream applications. I am aware of a >> number of API changes between 3.1.x and 3.3.1 and even between 3.3.0 and >> 3.3.1. You should use this as an opportunity to test out applications >> and be ready for it. >> >> (2) please check out the release notes and change log, find out if >> anything important should be included in 3.3.1. >> >> Please try the release and vote. The vote will run for 5 days until >> 2021/05/30 at 00:00 CST. >> >> Things I've verified: >> * all blocker issues targeting 3.3.1 have been resolved. >> * stable/evolving API changes between 3.3.0 and 3.3.1 are compatible. >> * LICENSE and NOTICE files checked >> * RELEASENOTES and CHANGELOG >> * rat check passed. >> >> >> Regards, >> Weichiu >> >>
