>
> I am an HBASE newbie so I apologise if I am being repetitious.
>
> Apologies also if this is not the right group. Am not sure if this may be
> more suited to 'dev' list.
>
Welcome, this is the right channel for this kind of questions.
> The solution offered by hbase-operator-tools - extraRegionsinMeta -
> offered hope. Once again, however, another problem has surfaced: this
> tools command for extra regions is incompatible with the hbase version we
> are running.
>
This command does not rely on any Master/RegionServer interface, so it
should not have any incompatibility issues. It only uses public client API
to cleanup meta table, so maybe you just need to build latest hbck2 master
branch version? It would work for any hbase 2, in theory, it could even
work with hbase 1, but that was never tested. Luckly, you got Stack's
attention, so yeah, if you confirm on his previous questions we might be
able to help with further directions.
Em ter., 15 de dez. de 2020 às 17:23, Stack <st...@duboce.net> escreveu:
> On Tue, Dec 15, 2020 at 7:46 AM Marc Hoppins <marc.hopp...@eset.sk> wrote:
>
> > Hi all,
> >
> > I am an HBASE newbie so I apologise if I am being repetitious.
> >
> > Apologies also if this is not the right group. Am not sure if this may be
> > more suited to 'dev' list. However,
> >
> > A problem question and a technical/wishlist question.
> >
> > Problem:
> >
> > I have inherited a problem with an HBASE table. The original issue may
> > have erupted due to a network outage. A table has 48 region in
> transition
> > operations, stuck that way for several weeks. A previous attempt to fix
> > things with hbck failed. An attempt to DISABLE then DROP the table also
> > failed. The four or five attempts to work the table ALSO now had stuck
> > procedures. Subsequent DFS and ZOO operations left the situation where
> > there was no data and no real table: just a 6K file in an empty
> structure.
> >
> >
> Which version of hbase, do you know Marc? (Look at the base of the master
> UI. It'll tell you. Sounds like it is an hbase-2.x).
>
> What is the name of the 6k file? (I am trying to understand what the file
> you are referencing is).
>
> The table directory was removed from hdfs and zookeeper?
>
>
>
> > When I got to the problem my knowledge of HBASE was nil. It is little
> > better than that now but anyway...
> >
> > Fortunately for me this is a testing/dev cluster. The 'owner' was content
> > that the table can be removed - and appeared to have already been
> done...of
> > a kind.
> >
> > Reading and reading of others' similar issues lead me to the point I also
> > was going to clean the HDFS data and ZK data for this table. I shut down
> > HBASE, cleaned HDFS and ZK node data, deleted the masterprocwals and
> > restarted HBASE.
> >
> > When all came up I was happy to see that the affected table appeared
> > nowhere and that the procedures had all disappeared.
> >
> > However, when I hopped to hbase master, even though no table of that name
> > existed, 48 regions were still in transition. Further research steered
> me
> > toward hbase:meta and sure enough, the references to the RITs lived
> happily
> > among other data for other tables.
> >
> >
> Do you have a 'Procedures and Locks' tab on your master home page? If so,
> what does it report? Is there an "HBCK Report" tab? If so, what does it
> show?
>
>
>
> > The solution offered by hbase-operator-tools - extraRegionsinMeta -
> > offered hope. Once again, however, another problem has surfaced: this
> > tools command for extra regions is incompatible with the hbase version we
> > are running.
> >
> > So...
> >
> > How can I remove the references to namespace:kaput_table from hbase:meta?
> >
> >
> > Sounds like an hbase-2.1.x or a 2.0.x.
>
> Will wait on your answers to the above... It might be a crass delete of
> each row from the hbase:meta table then restart (even then, if procedures
> in the procedure store, you may have to clear it again as you did above
> before the restart to purge the procedures as you don't have tooling to do
> it from cmdline... do you have an 'hbck2 bypass --override'?).
>
>
>
>
> > Technical:
> >
> > Is there to be any implementation of such a fix within HBASe itself where
> > table manipulation can be performed by Eg.,
> >
> > delete hbase:meta namespace
> > delete hbase:meta namespace:table
> >
> > or even
> >
> > scan hbase:meta filter = namespace:table | deleterow
> >
> > or some such?
> >
> >
> Scan doesn't return result (unfortunately) so you can assign to a shell
> variable; it dumps scan output2 on stdout/stderr.
>
> I'm sure there is a better way but something like below (don't laugh! and
> I've not really tried it so be careful):
>
> # Get rows for the BAD_TABLENAME from hbase:meta table
> $ echo 'scan "hbase:meta", {ROWPREFIXFILTER => "BAD_TABLENAME"}'| hbase
> shell > /tmp/out.txt
> # Need to get the row only. Above got rows and columns. We don't want to
> filter on column
> # because we do not know which columns are still in hbase:meta. Need to
> filter on table
> # again to cut the shell preamble and footer out. Unique the rows.
> $ cat /tmp/out.txt |awk '{print $1}'|grep BAD_TABLENAME | sort -u >
> /tmp/unique_rows.txt
> # Now you have rows to delete. For each in shell do... 'deleteall
> "ROW_X"'... Could
> # edit /tmp/unique_rows.txt and per for add .... deleteall " prefix and a "
> suffix.... to make
> # a file with lines of deleteall "ROW1", etc. then pass it to the shell:
> hbase shell /tmp/delete_rows.txt.
>
> But before messing w/ the above, lets get the answers to above questions.
> Thanks,
> S
>
>
>
>
>
>
>
>
> > Thanks
> >
> > M
> >
>
