[
https://issues.apache.org/jira/browse/HBASE-19287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16421429#comment-16421429
]
Abhishek Kulkarni commented on HBASE-19287:
-------------------------------------------
Dont see any error at regionserver other than warning
WARN [Close-WAL-Writer-3] asyncfs.FanOutOneBlockAsyncDFSOutputHelper: complete
file
/hbase/WALs/abhishekk3.pne.ven.veritas.com,16020,1522486816915/abhishekk3.pne.ven.veritas.com%2C16020%2C1522486816915.1522497620342
not finished, retry = 0
========
Sat Mar 31 05:00:13 EDT 2018 Starting regionserver on
abhishekk3.pne.ven.veritas.com
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 63395
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 63395
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
2018-03-31 05:00:16,504 INFO [main] regionserver.HRegionServer: STARTING
executorService HRegionServer
2018-03-31 05:00:16,505 INFO [main] util.VersionInfo: HBase 2.0.0-beta-2
2018-03-31 05:00:16,505 INFO [main] util.VersionInfo: Source code repository
git://ve0524/home/stack/hbase.git
revision=9e9b347d667e1fc6165c9f8ae5ae7052147e8895
2018-03-31 05:00:16,505 INFO [main] util.VersionInfo: Compiled by stack on Fri
Mar 2 13:29:06 PST 2018
2018-03-31 05:00:16,505 INFO [main] util.VersionInfo: From source with
checksum 07b3577d4d48c7a79e91a2d133e09db2
2018-03-31 05:00:16,888 INFO [main] util.ServerCommandLine: hbase.tmp.dir:
/tmp/hbase-root
2018-03-31 05:00:16,888 INFO [main] util.ServerCommandLine: hbase.rootdir:
hdfs://abhishekk1.pne.ven.veritas.com:54310/hbase
2018-03-31 05:00:16,888 INFO [main] util.ServerCommandLine:
hbase.cluster.distributed: true
2018-03-31 05:00:16,888 INFO [main] util.ServerCommandLine:
hbase.zookeeper.quorum:
abhishekk1.pne.ven.veritas.com,abhishekk2.pne.ven.veritas.com,abhishekk3.pne.ven.veritas.com
2018-03-31 05:00:16,889 INFO [main] util.ServerCommandLine:
env:PATH=/home/java/jdk1.8.0_131//bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/home/hadoop/hadoop-2.7.3//sbin:/home/hadoop/hadoop-2.7.3//bin:/home/hbase2/hbase-2.0.0-beta-2/bin
2018-03-31 05:00:16,889 INFO [main] util.ServerCommandLine:
env:HBASE_PID_DIR=/var/hbase/pids
2018-03-31 05:00:16,889 INFO [main] util.ServerCommandLine:
env:HBASE_REGIONSERVER_OPTS=-Djava.security.auth.login.config=/home/hbase2/hbase-2.0.0-beta-2/conf/hbaseregionserver-jaas.conf
2018-03-31 05:00:16,889 INFO [main] util.ServerCommandLine:
env:MAIL=/var/mail/root
2018-03-31 05:00:16,889 INFO [main] util.ServerCommandLine:
env:LD_LIBRARY_PATH=:/home/hadoop/hadoop-2.7.3//lib/native
2018-03-31 05:00:16,889 INFO [main] util.ServerCommandLine: env:LOGNAME=root
2018-03-31 05:00:16,889 INFO [main] util.ServerCommandLine:
env:HBASE_REST_OPTS=
2018-03-31 05:00:16,889 INFO [main] util.ServerCommandLine: env:PWD=/root
2018-03-31 05:00:16,889 INFO [main] util.ServerCommandLine:
env:HADOOP_INSTALL=/home/hadoop/hadoop-2.7.3/
2018-03-31 05:00:16,890 INFO [main] util.ServerCommandLine:
env:HBASE_ROOT_LOGGER=INFO,RFA
2018-03-31 05:00:16,890 INFO [main] util.ServerCommandLine:
env:LESSOPEN=||/usr/bin/lesspipe.sh %s
2018-03-31 05:00:16,890 INFO [main] util.ServerCommandLine: env:SHELL=/bin/bash
2018-03-31 05:00:16,890 INFO [main] util.ServerCommandLine:
env:HBASE_ENV_INIT=true
2018-03-31 05:00:16,890 INFO [main] util.ServerCommandLine:
env:HBASE_MASTER_OPTS=-Djava.security.auth.login.config=/home/hbase2/hbase-2.0.0-beta-2/conf/hbasemaster-jaas.conf
2018-03-31 05:00:16,890 INFO [main] util.ServerCommandLine:
env:HBASE_MANAGES_ZK=false
2018-03-31 05:00:16,890 INFO [main] util.ServerCommandLine:
env:HBASE_REGIONSERVERS=/home/hbase2/hbase-2.0.0-beta-2/conf/regionservers
2018-03-31 05:00:16,890 INFO [main] util.ServerCommandLine:
env:HADOOP_HOME=/home/hadoop/hadoop-2.7.3/
2018-03-31 05:00:16,890 INFO [main] util.ServerCommandLine:
env:HBASE_NICENESS=0
2018-03-31 05:00:16,890 INFO [main] util.ServerCommandLine:
env:HBASE_OPTS=-Djava.security.auth.login.config=/home/hbase2/hbase-2.0.0-beta-2/conf/hbaseclient-jass.conf
-Djava.security.auth.login.config=/home/hbase2/hbase-2.0.0-beta-2/conf/hbaseregionserver-jaas.conf
-Dhbase.log.dir=/home/hbase2/hbase-2.0.0-beta-2/logs
-Dhbase.log.file=hbase-root-regionserver-abhishekk3.pne.ven.veritas.com.log
-Dhbase.home.dir=/home/hbase2/hbase-2.0.0-beta-2 -Dhbase.id.str=root
-Dhbase.root.logger=INFO,RFA
-Djava.library.path=/home/hadoop/hadoop-2.7.3//lib/native
-Dhbase.security.logger=INFO,RFAS
2018-03-31 05:00:16,890 INFO [main] util.ServerCommandLine:
env:HBASE_SECURITY_LOGGER=INFO,RFAS
2018-03-31 05:00:16,890 INFO [main] util.ServerCommandLine: env:SHLVL=3
2018-03-31 05:00:16,890 INFO [main] util.ServerCommandLine:
env:ZOOKEEPER_HOME=/home/hbas2/zookeeper/zookeeper-3.5.3-beta
2018-03-31 05:00:16,890 INFO [main] util.ServerCommandLine:
env:HBASE_LOGFILE=hbase-root-regionserver-abhishekk3.pne.ven.veritas.com.log
2018-03-31 05:00:16,890 INFO [main] util.ServerCommandLine:
env:HBASE_ZOOKEEPER_OPTS= -Xdebug
-Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=8073
2018-03-31 05:00:16,890 INFO [main] util.ServerCommandLine:
env:JAVA_HOME=/home/java/jdk1.8.0_131/
2018-03-31 05:00:16,890 INFO [main] util.ServerCommandLine:
env:XFILESEARCHPATH=/usr/dt/app-defaults/%L/Dt
2018-03-31 05:00:16,890 INFO [main] util.ServerCommandLine:
env:LANG=en_US.UTF-8
2018-03-31 05:00:16,890 INFO [main] util.ServerCommandLine:
env:XDG_SESSION_ID=6
2018-03-31 05:00:16,890 INFO [main] util.ServerCommandLine:
env:YARN_HOME=/home/hadoop/hadoop-2.7.3/
2018-03-31 05:00:16,890 INFO [main] util.ServerCommandLine:
env:HADOOP_HDFS_HOME=/home/hadoop/hadoop-2.7.3/
2018-03-31 05:00:16,890 INFO [main] util.ServerCommandLine:
env:HADOOP_MAPRED_HOME=/home/hadoop/hadoop-2.7.3/
2018-03-31 05:00:16,891 INFO [main] util.ServerCommandLine:
env:HADOOP_COMMON_HOME=/home/hadoop/hadoop-2.7.3/
2018-03-31 05:00:16,891 INFO [main] util.ServerCommandLine: env:HADOOP_OPTS=
-Djava.library.path=/home/hadoop/hadoop-2.7.3//lib/native
2018-03-31 05:00:16,891 INFO [main] util.ServerCommandLine:
env:HBASE_IDENT_STRING=root
2018-03-31 05:00:16,891 INFO [main] util.ServerCommandLine:
env:HBASE_ZNODE_FILE=/var/hbase/pids/hbase-root-regionserver.znode
2018-03-31 05:00:16,891 INFO [main] util.ServerCommandLine:
env:SSH_CLIENT=10.210.62.28 48404 22
2018-03-31 05:00:16,891 INFO [main] util.ServerCommandLine:
env:HBASE_LOG_PREFIX=hbase-root-regionserver-abhishekk3.pne.ven.veritas.com
2018-03-31 05:00:16,891 INFO [main] util.ServerCommandLine:
env:HBASE_LOG_DIR=/home/hbase2/hbase-2.0.0-beta-2/logs
2018-03-31 05:00:16,891 INFO [main] util.ServerCommandLine: env:USER=root
2018-03-31 05:00:16,891 INFO [main] util.ServerCommandLine:
me/hbase2/hbase-2.0.0-beta-2/lib/protobuf-java-2.5.0.jar:/home/hbase2/hbase-2.0.0-beta-2/lib/slf4j-api-1.7.25.jar:/home/hbase2/hbase-2.0.0-beta-2/lib/snappy-java-1.0.5.jar:/home/hbase2/hbase-2.0.0-beta-2/lib/spymemcached-2.12.2.jar:/home/hbase2/hbase-2.0.0-beta-2/lib/validation-api-1.1.0.Final.jar:/home/hbase2/hbase-2.0.0-beta-2/lib/xmlenc-0.52.jar:/home/hbase2/hbase-2.0.0-beta-2/lib/xz-1.0.jar:/home/hbase2/hbase-2.0.0-beta-2/lib/zookeeper-3.4.10.jar:/home/hadoop/hadoop-2.7.3/etc/hadoop:/home/hadoop/hadoop-2.7.3//share/hadoop/common/lib/*:/home/hadoop/hadoop-2.7.3//share/hadoop/common/*:/home/hadoop/hadoop-2.7.3//share/hadoop/hdfs:/home/hadoop/hadoop-2.7.3//share/hadoop/hdfs/lib/*:/home/hadoop/hadoop-2.7.3//share/hadoop/hdfs/*:/home/hadoop/hadoop-2.7.3/share/hadoop/yarn/lib/*:/home/hadoop/hadoop-2.7.3/share/hadoop/yarn/*:/home/hadoop/hadoop-2.7.3//share/hadoop/mapreduce/lib/*:/home/hadoop/hadoop-2.7.3//share/hadoop/mapreduce/*:/home/hadoop/hadoop-2.7.3//contrib/capacity-scheduler/*.jar
2018-03-31 05:00:16,891 INFO [main] util.ServerCommandLine:
env:SSH_CONNECTION=10.210.62.28 48404 10.210.62.30 22
2018-03-31 05:00:16,891 INFO [main] util.ServerCommandLine:
env:HBASE_AUTOSTART_FILE=/var/hbase/pids/hbase-root-regionserver.autostart
2018-03-31 05:00:16,891 INFO [main] util.ServerCommandLine:
env:NLSPATH=/usr/dt/lib/nls/msg/%L/%N.cat
2018-03-31 05:00:16,891 INFO [main] util.ServerCommandLine:
env:HADOOP_COMMON_LIB_NATIVE_DIR=/home/hadoop/hadoop-2.7.3//lib/native
2018-03-31 05:00:16,891 INFO [main] util.ServerCommandLine:
env:XDG_RUNTIME_DIR=/run/user/0
2018-03-31 05:00:16,891 INFO [main] util.ServerCommandLine:
env:HBASE_THRIFT_OPTS=
2018-03-31 05:00:16,891 INFO [main] util.ServerCommandLine:
env:HBASE_HOME=/home/hbase2/hbase-2.0.0-beta-2
2018-03-31 05:00:16,892 INFO [main] util.ServerCommandLine: env:HOME=/root
2018-03-31 05:00:16,892 INFO [main] util.ServerCommandLine:
env:MALLOC_ARENA_MAX=4
2018-03-31 05:00:16,893 INFO [main] util.ServerCommandLine: vmName=Java
HotSpot(TM) 64-Bit Server VM, vmVendor=Oracle Corporation, vmVersion=25.131-b11
2018-03-31 05:00:16,893 INFO [main] util.ServerCommandLine:
vmInputArguments=[-Dproc_regionserver, -XX:OnOutOfMemoryError=kill -9 %p,
-Djava.security.auth.login.config=/home/hbase2/hbase-2.0.0-beta-2/conf/hbaseclient-jass.conf,
-Djava.security.auth.login.config=/home/hbase2/hbase-2.0.0-beta-2/conf/hbaseregionserver-jaas.conf,
-Dhbase.log.dir=/home/hbase2/hbase-2.0.0-beta-2/logs,
-Dhbase.log.file=hbase-root-regionserver-abhishekk3.pne.ven.veritas.com.log,
-Dhbase.home.dir=/home/hbase2/hbase-2.0.0-beta-2, -Dhbase.id.str=root,
-Dhbase.root.logger=INFO,RFA,
-Djava.library.path=/home/hadoop/hadoop-2.7.3//lib/native,
-Dhbase.security.logger=INFO,RFAS]
2018-03-31 05:00:17,279 INFO [main] metrics.MetricRegistries: Loaded
MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl
2018-03-31 05:00:17,444 INFO [main] regionserver.RSRpcServices: hostname is
configured to be abhishekk3.pne.ven.veritas.com
2018-03-31 05:00:17,555 INFO [main] regionserver.RSRpcServices:
regionserver/abhishekk3:16020 server-side Connection retries=45
2018-03-31 05:00:17,570 INFO [main] ipc.RpcExecutor: Instantiated
default.FPBQ.Fifo with queueClass=class
java.util.concurrent.LinkedBlockingQueue; numCallQueues=3, maxQueueLength=300,
handlerCount=30
2018-03-31 05:00:17,572 INFO [main] ipc.RpcExecutor: Instantiated
priority.FPBQ.Fifo with queueClass=class
java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=300,
handlerCount=20
2018-03-31 05:00:17,572 INFO [main] ipc.RpcExecutor: Instantiated
replication.FPBQ.Fifo with queueClass=class
java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=300,
handlerCount=3
2018-03-31 05:00:17,675 INFO [main] ipc.RpcServerFactory: Creating
org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService,
hbase.pb.AdminService
2018-03-31 05:00:17,892 INFO [main] io.ByteBufferPool: Created with
bufferSize=64 KB and maxPoolSize=1.88 KB
2018-03-31 05:00:17,978 INFO [main] ipc.NettyRpcServer: Bind to
/10.210.62.30:16020
2018-03-31 05:00:18,194 INFO [main] security.UserGroupInformation: Login
successful for user hbase/[email protected]
using keytab file /etc/security/keytab/hbase-abhishekk3.service.keytab
2018-03-31 05:00:18,197 INFO [main] hfile.CacheConfig: Allocating onheap
LruBlockCache size=1.38 GB, blockSize=64 KB
2018-03-31 05:00:18,204 INFO [main] hfile.CacheConfig: Created cacheConfig:
blockCache=LruBlockCache\{blockCount=0, currentSize=1.03 MB, freeSize=1.38 GB,
maxSize=1.38 GB, heapSize=1.03 MB, minSize=1.31 GB, minFactor=0.95,
multiSize=670.51 MB, multiFactor=0.5, singleSize=335.26 MB, singleFactor=0.25},
cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false,
cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false,
prefetchOnOpen=false
2018-03-31 05:00:18,205 INFO [main] hfile.CacheConfig: Created cacheConfig:
blockCache=LruBlockCache\{blockCount=0, currentSize=1.03 MB, freeSize=1.38 GB,
maxSize=1.38 GB, heapSize=1.03 MB, minSize=1.31 GB, minFactor=0.95,
multiSize=670.51 MB, multiFactor=0.5, singleSize=335.26 MB, singleFactor=0.25},
cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false,
cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false,
prefetchOnOpen=false
2018-03-31 05:00:18,799 INFO [main] fs.HFileSystem: Added intercepting call to
namenode#getBlockLocations so can do block reordering using class
org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks
2018-03-31 05:00:18,802 INFO [main] fs.HFileSystem: Added intercepting call to
namenode#getBlockLocations so can do block reordering using class
org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks
2018-03-31 05:00:18,851 INFO [main] zookeeper.RecoverableZooKeeper: Process
identifier=regionserver:16020 connecting to ZooKeeper
ensemble=abhishekk1.pne.ven.veritas.com:2181,abhishekk2.pne.ven.veritas.com:2181,abhishekk3.pne.ven.veritas.com:2181
2018-03-31 05:00:18,856 INFO [main] zookeeper.ZooKeeper: Client
environment:zookeeper.version=3.4.10-39d3a4f269333c922ed3db283be479f9deacaa0f,
built on 03/23/2017 10:13 GMT
2018-03-31 05:00:18,856 INFO [main] zookeeper.ZooKeeper: Client
environment:host.name=abhishekk3.pne.ven.veritas.com
2018-03-31 05:00:18,856 INFO [main] zookeeper.ZooKeeper: Client
environment:java.version=1.8.0_131
2018-03-31 05:00:18,856 INFO [main] zookeeper.ZooKeeper: Client
environment:java.vendor=Oracle Corporation
2018-03-31 05:00:18,857 INFO [main] zookeeper.ZooKeeper: Client
environment:java.home=/home/java/jdk1.8.0_131/jre
2018-03-31 05:00:18,857 INFO [main] zookeeper.ZooKeeper:
/share/hadoop/mapreduce/lib/junit-4.11.jar:/home/hadoop/hadoop-2.7.3//share/hadoop/mapreduce/lib/hamcrest-core-1.3.jar:/home/hadoop/hadoop-2.7.3//share/hadoop/mapreduce/hadoop-mapreduce-client-core-2.7.3.jar:/home/hadoop/hadoop-2.7.3//share/hadoop/mapreduce/hadoop-mapreduce-client-common-2.7.3.jar:/home/hadoop/hadoop-2.7.3//share/hadoop/mapreduce/hadoop-mapreduce-client-shuffle-2.7.3.jar:/home/hadoop/hadoop-2.7.3//share/hadoop/mapreduce/hadoop-mapreduce-client-app-2.7.3.jar:/home/hadoop/hadoop-2.7.3//share/hadoop/mapreduce/hadoop-mapreduce-client-hs-2.7.3.jar:/home/hadoop/hadoop-2.7.3//share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.7.3.jar:/home/hadoop/hadoop-2.7.3//share/hadoop/mapreduce/hadoop-mapreduce-client-hs-plugins-2.7.3.jar:/home/hadoop/hadoop-2.7.3//share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.3.jar:/home/hadoop/hadoop-2.7.3//share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.7.3-tests.jar:/home/hadoop/hadoop-2.7.3//contrib/capacity-scheduler/*.jar
2018-03-31 05:00:18,857 INFO [main] zookeeper.ZooKeeper: Client
environment:java.library.path=/home/hadoop/hadoop-2.7.3//lib/native
2018-03-31 05:00:18,857 INFO [main] zookeeper.ZooKeeper: Client
environment:java.io.tmpdir=/tmp
2018-03-31 05:00:18,857 INFO [main] zookeeper.ZooKeeper: Client
environment:java.compiler=<NA>
2018-03-31 05:00:18,857 INFO [main] zookeeper.ZooKeeper: Client
environment:os.name=Linux
2018-03-31 05:00:18,857 INFO [main] zookeeper.ZooKeeper: Client
environment:os.arch=amd64
2018-03-31 05:00:18,857 INFO [main] zookeeper.ZooKeeper: Client
environment:os.version=3.10.0-514.el7.x86_64
2018-03-31 05:00:18,857 INFO [main] zookeeper.ZooKeeper: Client
environment:user.name=root
2018-03-31 05:00:18,857 INFO [main] zookeeper.ZooKeeper: Client
environment:user.home=/root
2018-03-31 05:00:18,857 INFO [main] zookeeper.ZooKeeper: Client
environment:user.dir=/root
2018-03-31 05:00:18,858 INFO [main] zookeeper.ZooKeeper: Initiating client
connection,
connectString=abhishekk1.pne.ven.veritas.com:2181,abhishekk2.pne.ven.veritas.com:2181,abhishekk3.pne.ven.veritas.com:2181
sessionTimeout=10000000
watcher=org.apache.hadoop.hbase.zookeeper.PendingWatcher@7c6189d5
2018-03-31 05:00:18,959 INFO
[main-SendThread(abhishekk2.pne.ven.veritas.com:2181)] zookeeper.Login: Client
successfully logged in.
2018-03-31 05:00:18,960 INFO [Thread-6] zookeeper.Login: TGT refresh thread
started.
2018-03-31 05:00:18,964 INFO [Thread-6] zookeeper.Login: TGT valid starting
at: Sat Mar 31 04:56:53 EDT 2018
2018-03-31 05:00:18,964 INFO [Thread-6] zookeeper.Login: TGT expires:
Sun Apr 01 04:56:53 EDT 2018
2018-03-31 05:00:18,964 INFO [Thread-6] zookeeper.Login: TGT refresh sleeping
until: Sun Apr 01 00:14:20 EDT 2018
2018-03-31 05:00:18,964 INFO
[main-SendThread(abhishekk2.pne.ven.veritas.com:2181)] client.ZooKeeperSasl
> master hangs forever if RecoverMeta send assign meta region request to target
> server fail
> -----------------------------------------------------------------------------------------
>
> Key: HBASE-19287
> URL: https://issues.apache.org/jira/browse/HBASE-19287
> Project: HBase
> Issue Type: Bug
> Components: proc-v2
> Affects Versions: 2.0.0
> Reporter: Yi Liang
> Assignee: Yi Liang
> Priority: Major
> Fix For: 2.0.0-beta-1, 2.0.0
>
> Attachments: HBASE-19287-master-v3.patch,
> HBASE-19287-master-v3.patch, HBASE-19287-master-v4.patch,
> hbase-19287-master-v2.patch, master.patch
>
>
> 2017-11-10 19:26:56,019 INFO [ProcExecWrkr-1]
> procedure.RecoverMetaProcedure: pid=138,
> state=RUNNABLE:RECOVER_META_ASSIGN_REGIONS; RecoverMetaProcedure
> failedMetaServer=null, splitWal=true; Retaining meta assignment to
> server=hadoop-slave1.hadoop,16020,1510341981454
> 2017-11-10 19:26:56,029 INFO [ProcExecWrkr-1] procedure2.ProcedureExecutor:
> Initialized subprocedures=[{pid=139, ppid=138,
> state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:meta,
> region=1588230740, target=hadoop-slave1.hadoop,16020,1510341981454}]
> 2017-11-10 19:26:56,067 INFO [ProcExecWrkr-2]
> procedure.MasterProcedureScheduler: pid=139, ppid=138,
> state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:meta,
> region=1588230740, target=hadoop-slave1.hadoop,16020,1510341981454 hbase:meta
> hbase:meta,,1.1588230740
> 2017-11-10 19:26:56,071 INFO [ProcExecWrkr-2] assignment.AssignProcedure:
> Start pid=139, ppid=138, state=RUNNABLE:REGION_TRANSITION_QUEUE;
> AssignProcedure table=hbase:meta, region=1588230740,
> target=hadoop-slave1.hadoop,16020,1510341981454; rit=OFFLINE,
> location=hadoop-slave1.hadoop,16020,1510341981454; forceNewPlan=false,
> retain=false
> 2017-11-10 19:26:56,224 INFO [ProcExecWrkr-4] zookeeper.MetaTableLocator:
> Setting hbase:meta (replicaId=0) location in ZooKeeper as
> hadoop-slave2.hadoop,16020,1510341988652
> 2017-11-10 19:26:56,230 INFO [ProcExecWrkr-4]
> assignment.RegionTransitionProcedure: Dispatch pid=139, ppid=138,
> state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=hbase:meta,
> region=1588230740, target=hadoop-slave1.hadoop,16020,1510341981454;
> rit=OPENING, location=hadoop-slave2.hadoop,16020,1510341988652
> 2017-11-10 19:26:56,382 INFO [ProcedureDispatcherTimeoutThread]
> procedure.RSProcedureDispatcher: Using procedure batch rpc execution for
> serverName=hadoop-slave2.hadoop,16020,1510341988652 version=2097152
> 2017-11-10 19:26:57,542 INFO [main-EventThread]
> zookeeper.RegionServerTracker: RegionServer ephemeral node deleted,
> processing expiration [hadoop-slave2.hadoop,16020,1510341988652]
> 2017-11-10 19:26:57,543 INFO [main-EventThread] master.ServerManager: Master
> doesn't enable ServerShutdownHandler during initialization, delay expiring
> server hadoop-slave2.hadoop,16020,1510341988652
> 2017-11-10 19:26:58,875 INFO
> [RpcServer.default.FPBQ.Fifo.handler=29,queue=2,port=16000]
> master.ServerManager: Registering
> server=hadoop-slave1.hadoop,16020,1510342016106
> 2017-11-10 19:27:05,832 INFO
> [RpcServer.default.FPBQ.Fifo.handler=29,queue=2,port=16000]
> master.ServerManager: Registering
> server=hadoop-slave2.hadoop,16020,1510342023184
> 2017-11-10 19:27:05,832 INFO
> [RpcServer.default.FPBQ.Fifo.handler=29,queue=2,port=16000]
> master.ServerManager: Triggering server recovery; existingServer
> hadoop-slave2.hadoop,16020,1510341988652 looks stale, new
> server:hadoop-slave2.hadoop,16020,1510342023184
> 2017-11-10 19:27:05,832 INFO
> [RpcServer.default.FPBQ.Fifo.handler=29,queue=2,port=16000]
> master.ServerManager: Master doesn't enable ServerShutdownHandler during
> initialization, delay expiring server hadoop-slave2.hadoop,16020,1510341988652
> 2017-11-10 19:27:49,815 INFO
> [RpcServer.default.FPBQ.Fifo.handler=29,queue=2,port=16000]
> client.RpcRetryingCallerImpl: tarted=38594 ms ago, cancelled=false,
> msg=org.apache.hadoop.hbase.NotServingRegionException: hbase:meta,,1 is not
> online on hadoop-slave2.hadoop,16020,1510342023184
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:3290)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegion(RSRpcServices.java:1370)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2401)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41544)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:406)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:133)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:278)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:258)
> row 'hbase:namespace' on table 'hbase:meta' at
> region=hbase:meta,,1.1588230740,
> hostname=hadoop-slave2.hadoop,16020,1510341988652, seqNum=0
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
