> What are your challenges upgrading to Hadoop 3 ? It's a major release upgradation from hadoop 2 to hadoop 3. we upstreams will need some time to test the stability and performance, so I think the HBase will not upgrade to hadoop 3 so quick in short time. Backporting the ByteBuffer pread to hadoop 2.9 is more suitable for now (it's a minor feature, should be OK to port in branch-2.9). I believe in future HBase release we will include the hadoop 3
> Did you verify if your problem statement valid, if you use Ozone API ? We have some work about the object-store api [1]. but I'm not quite sure how will it affect the GC. will take a look about that. Thanks for your reply. [1]. https://issues.apache.org/jira/browse/HBASE-22149 On Fri, Jun 28, 2019 at 7:28 AM Sree Vaddi <sree_at_ch...@yahoo.com> wrote: > What are your challenges upgrading to Hadoop 3 ? > > Did you verify if your problem statement valid, if you use Ozone API ? > > Sent from Yahoo Mail on Android > <https://go.onelink.me/107872968?pid=InProduct&c=Global_Internal_YGrowth_AndroidEmailSig__AndroidUsers&af_wl=ym&af_sub1=Internal&af_sub2=Global_YGrowth&af_sub3=EmailSignature> > > On Thu, Jun 27, 2019 at 9:20 AM, OpenInx > <open...@gmail.com> wrote: > Dear HDFS Team: > > I'm apache HBase member and working on HBASE-21879 [1]: it's a feature > which help to optimize the > HBase GC issues, we redesigned the HBase read(Scan/Get) path by reading > HDFS block into offheap > ByteBuffers and it had great performance improvement in our benchmark test > [2]. While the offheap > HDFS block reading highly depend on the HDFS ByteBuffer interfaces, such > as: ByteBuffer stream > read and ByteBuffer positional read. > > The problem is ByteBuffer pread interface: we've finished the HDFS-3246[3] > for Hadoop 3.3.0 version. but > the Apache HBase is depending on Hadoop 2.8.5 version, and seems will not > upgrade the dependency to > Hadoop 3.3.0 soon. We have to backport HDFS-3246 to Hadoop branch-2.9 so > that in the next hadoop > 2.9.3 can include it , and our Apache HBase can upgrade the dependency to > 2.9.3. > > Now all the patches of HBASE-21879 have been merged into HBase master > branch. If we cannot upgrade to > the Hadoop which include the ByteBuffer pread implementation, then I > believe almost all of work from HBASE-21879 > will worth nothing because we still read heap block and still high GC > pressure in some special case ( such as > cacheHitRatio is not very high). That will be quite frustrating for us. > > Actually, we have been pushing the backport for the past few weeks. our > XiaoMi HDFS team have prepared the > backport patches, but seems the guys we ping didn't have the bandwidth to > help review the patches. we > expect the backport can be included in Hadoop 2.9.3 so that our upstream > can benefit from this as soon as > possible. > > If any HDFS committer or PMC have some bandwidth to review the backport > patches, please give us a hand. > The Apache HBase team really need your help. > > The ByteBuffer pread backport issues are: > https://issues.apache.org/jira/browse/HDFS-14585 > https://issues.apache.org/jira/browse/HDFS-14483 > > Thank you very much. > > [1]. https://issues.apache.org/jira/browse/HBASE-21879 > [2]. https://www.mail-archive.com/dev@hbase.apache.org/msg55335.html > [3]. https://issues.apache.org/jira/browse/HDFS-3246 > >
