HBase Master hangs on startup during upgrade from 2.2.5 to 2.5.7

2024-04-20 Thread Udo Offermann
Hello everyone, We are upgrading our Hadoop/HBase cluster from Hadoop 2.8.5 & HBase 2.2.5 to Hadoop 3.3.6 & HBase 2.5.7 The Hadoop upgrade worked well, but unfortunately we have problems with the Hbase upgrade, because the master hangs on startup inside the „Starting assignment manger“ task. Aft

Re: HBase Master hangs on startup during upgrade from 2.2.5 to 2.5.7

2024-04-20 Thread Duo Zhang
What is the jstack result for HMaster while hanging? Wait on the namespace table online or meta table online? Udo Offermann 于2024年4月20日周六 19:43写道: > > Hello everyone, > > We are upgrading our Hadoop/HBase cluster from Hadoop 2.8.5 & HBase 2.2.5 > to Hadoop 3.3.6 & HBase 2.5.7 > > The Hadoop upgra

Re: HBase Master hangs on startup during upgrade from 2.2.5 to 2.5.7

2024-04-20 Thread Udo Offermann
I do have the dump File from the web ui. I can sende it all or you Tell me threads you are interessted in. Fortunately they all have meaningfull named. 张铎(Duo Zhang) schrieb am Sa., 20. Apr. 2024, 14:13: > What is the jstack result for HMaster while hanging? Wait on the > namespace table online

Re: HBase Master hangs on startup during upgrade from 2.2.5 to 2.5.7

2024-04-20 Thread Duo Zhang
Just post it somewhere so we can check it. Udo Offermann 于2024年4月20日周六 20:25写道: > > I do have the dump File from the web ui. I can sende it all or you Tell me > threads you are interessted in. Fortunately they all have meaningfull named. > > 张铎(Duo Zhang) schrieb am Sa., 20. Apr. 2024, 14:13: >

Re: HBase Master hangs on startup during upgrade from 2.2.5 to 2.5.7

2024-04-20 Thread Udo Offermann
Master status for masterserver.gmd9.intern,16000,1713515965162 as of Fri Apr 19 10:55:22 CEST 2024 Version Info: === HBase 2.5.7 Source code repository git://buildbox.localdomain/home/apurtell/tmp/RM/hbase revision=6788f98356dd70b4a7ff766ea7

Re: HBase Master hangs on startup during upgrade from 2.2.5 to 2.5.7

2024-04-20 Thread Duo Zhang
OK, it was waitForMetaOnline. Maybe the problem is that you do have some correct procedures before upgrading, like ServerCrashProcedure, but then you delete all the procedure wals so the ServerCrashProcedure is also gone, so meta can never be online. Please check the /hbase/meta-region-server zno

Re: HBase Master hangs on startup during upgrade from 2.2.5 to 2.5.7

2024-04-20 Thread Udo Offermann
Thank you, I can check on Monday. This is the upgrade of the test system and serves as training for the upgrade of the production system. What do we need to do to prevent this problem? We had some problems starting zookeeper after the upgrade and I had to start it with "zookeeper.snapshot.tru