Just reporting back on mailing list that i was able to fix the issue by a lot of manual work. This problem happened because we forgot to run "upgrade -execute" after installing hbase0.98 and before starting 0.98. If you dont run "upgrade -execute" then user gets and error regarding "hbase.version". I deleted that file and then HBase came up. But since, i deleted hbase.versions file so HBase didnt do any migration of my 0.94 data.
Here is the high level stuff that i did to fix this issue(NOTE: these steps may vary as per cluster setup): In ZK i did following stuff: Deleted all znodes under /hbase/table Recursively Deleted /hbase/table-lock znode delete /hbase/meta-region-server delete /hbase/region-in-transition delete /hbase/balancer rmr /hbase/namespace rmr /hbase/replication rmr /hbase/recovering-regions rmr /hbase/splitWAL In HDFS: I removed /hbase/data directory and moved around some other related directory. Updated the hbase.version file to 0.94 version file. Then i ran "upgrade -execute" and it ran successfully. Started the cluster and i was able to see all the tables along with their data. Thanks Stack for your help. ~Anil On Mon, Jul 28, 2014 at 11:01 PM, anil gupta <[email protected]> wrote: > Please find my reply inline. > > > On Mon, Jul 28, 2014 at 10:10 PM, Stack <[email protected]> wrote: > >> On Mon, Jul 28, 2014 at 8:39 PM, anil gupta <[email protected]> >> wrote: >> >> > Please find my reply inline. >> > >> > >> > On Mon, Jul 28, 2014 at 6:09 PM, Stack <[email protected]> wrote: >> > >> > > On Mon, Jul 28, 2014 at 5:45 PM, anil gupta <[email protected]> >> > wrote: >> > > >> > > > Hi Stack, >> > > > >> > > > I basically did full table scan for hbase:meta by running this(since >> > > > cluster has 2-3 regions only): scan 'hbase:meta', {RAW => true, >> > VERSIONS >> > > => >> > > > 10} >> > > > Output of above query shows rows for "hbase:namespace" and "dummy" >> > table. >> > > > So, it seems like hbase:meta doesn't have any old data. >> > > > >> > > > >> > > Yeah. Check size of the hbase:meta region. If small, yeah, may not >> have >> > > the above. If you look around in hdfs, you see an old .META. dir? >> > > >> > > Anil: Yes, it looks like this dir "/hbase/.META." has data of meta >> table >> > before upgrade to 0.98. Are there any non-backward compatible changes in >> > META? >> > Is it possible to just import the HFile of 0.94 in hbase:meta table? It >> > seems like directory structure under hdfs has changes in 0.98. I am >> unable >> > to find directories for a table "dummy" i created in 0.98. >> > >> > >> >> I think what happened is that because you failed run the upgrade script, >> 0.98 tried to start over a 0.94 and complained about the hbase.version >> file >> it found because it had not be protobuf'd. You did what I would have >> done, >> 'fixed it', by removing the obstacle only when hbase does not find an >> hbase.version, it thinks it all a fresh install. >> > Anil: Well, deleting hbase.version turned out be a hack that proved > costly. I take back my word 'fixed it' :) > >> >> >> Without knowing much more, if you put back a 0.94 era hbase.version in >> place of the one written by 0.98, then you ran the upgrade script, I >> wonder >> if that'd fix it? >> > Anil: Gonna try this soon and report back. > >> >> >> >> > > > Our sysadmin team is still trying to locate all the logs. However, i >> > > > remember one error that i resolved. I was related to >> "hbase.versions" >> > > file. >> > > > It was some complaint about Protbouf version. I deleted >> hbase.versions >> > > file >> > > > and then HBase came up. >> > > >> > > >> > > Hmm. So, it probably thought this a new install and just created the >> new >> > > layout. Is the old 0.94 data is adjacent to the new layout? >> > > >> > Anil: Yes, that's what i feel. The upgrade happened as it was a fresh >> > install. Does it seems like a bug in upgrade? >> > >> > >> Well, if you ended up in this state, yeah, there is a problem with it. >> >> St.Ack >> > > > > -- > Thanks & Regards, > Anil Gupta > -- Thanks & Regards, Anil Gupta
