What are your challenges upgrading to Hadoop 3 ? Did you verify if your problem statement valid, if you use Ozone API ?
Sent from Yahoo Mail on Android On Thu, Jun 27, 2019 at 9:20 AM, OpenInx<open...@gmail.com> wrote: Dear HDFS Team: I'm apache HBase member and working on HBASE-21879 [1]: it's a feature which help to optimize the HBase GC issues, we redesigned the HBase read(Scan/Get) path by reading HDFS block into offheap ByteBuffers and it had great performance improvement in our benchmark test [2]. While the offheap HDFS block reading highly depend on the HDFS ByteBuffer interfaces, such as: ByteBuffer stream read and ByteBuffer positional read. The problem is ByteBuffer pread interface: we've finished the HDFS-3246[3] for Hadoop 3.3.0 version. but the Apache HBase is depending on Hadoop 2.8.5 version, and seems will not upgrade the dependency to Hadoop 3.3.0 soon. We have to backport HDFS-3246 to Hadoop branch-2.9 so that in the next hadoop 2.9.3 can include it , and our Apache HBase can upgrade the dependency to 2.9.3. Now all the patches of HBASE-21879 have been merged into HBase master branch. If we cannot upgrade to the Hadoop which include the ByteBuffer pread implementation, then I believe almost all of work from HBASE-21879 will worth nothing because we still read heap block and still high GC pressure in some special case ( such as cacheHitRatio is not very high). That will be quite frustrating for us. Actually, we have been pushing the backport for the past few weeks. our XiaoMi HDFS team have prepared the backport patches, but seems the guys we ping didn't have the bandwidth to help review the patches. we expect the backport can be included in Hadoop 2.9.3 so that our upstream can benefit from this as soon as possible. If any HDFS committer or PMC have some bandwidth to review the backport patches, please give us a hand. The Apache HBase team really need your help. The ByteBuffer pread backport issues are: https://issues.apache.org/jira/browse/HDFS-14585 https://issues.apache.org/jira/browse/HDFS-14483 Thank you very much. [1]. https://issues.apache.org/jira/browse/HBASE-21879 [2]. https://www.mail-archive.com/dev@hbase.apache.org/msg55335.html [3]. https://issues.apache.org/jira/browse/HDFS-3246
