Hi Vergil, Thanks for sharing the information. Can you please share the complete logs from where you have collected the info? Thanks, Rajeshbabu.
On Mon, Nov 6, 2017 at 10:08 AM, vergil <vergilc...@foxmail.com> wrote: > Thank for your reply. > > Here is My test environment： > hbase:1.2.5 > phoenix:4.12 > > Phoenix table with local index。 > > The region will wait for a long time when the region split。 > *The region server webapp page show below:* > Start TimeDescriptionStateStatus > Mon Nov 06 17:49:27 CST 2017 Closing region TEST_TABLE_LOCAL,\x01, > 1509961563238.9070a4f2ace4516f985c6c9e31427eee. RUNNING (since 1mins, > 46sec ago) Finished memstore flush of ~39.94 MB/41879888, > currentsize=14.31 MB/15002912 for region TEST_TABLE_LOCAL,\x01, > 1509961563238.9070a4f2ace4516f985c6c9e31427eee. in 856ms, > sequenceid=41861, compaction requested=true (since 1mins, 45sec ago) > *The region server wait for a long time and log show below :* > 2017-11-06 18:02:06,450 INFO > [B.defaultRpcServer.handler=27,queue=0,port=16020] > index.PhoenixIndexFailurePolicy: Successfully disabled index > TEST_INDEX_LOCAL due to an exception while writing updates. > org.apache.phoenix.hbase.index.exception.MultiIndexWriteFailureException: > Failed to write to multiple index tables > at org.apache.phoenix.hbase.index.write.TrackingParallelWriterIndexCom > mitter.write(TrackingParallelWriterIndexCommitter.java:229) > at org.apache.phoenix.hbase.index.write.IndexWriter.write( > IndexWriter.java:193) > at org.apache.phoenix.hbase.index.write.IndexWriter. > writeAndKillYourselfOnFailure(IndexWriter.java:154) > at org.apache.phoenix.hbase.index.write.IndexWriter. > writeAndKillYourselfOnFailure(IndexWriter.java:143) > at org.apache.phoenix.hbase.index.Indexer.doPostWithExceptions(Indexer. > java:653) > at org.apache.phoenix.hbase.index.Indexer.doPost(Indexer.java:610) > at org.apache.phoenix.hbase.index.Indexer.postBatchMutateIndispensably( > Indexer.java:593) > at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$37.call( > RegionCoprocessorHost.java:1034) > at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$ > RegionOperation.call(RegionCoprocessorHost.java:1673) > at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost. > execOperation(RegionCoprocessorHost.java:1749) > at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost. > execOperation(RegionCoprocessorHost.java:1705) > at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost. > postBatchMutateIndispensably(RegionCoprocessorHost.java:1030) > at org.apache.hadoop.hbase.regionserver.HRegion. > doMiniBatchMutation(HRegion.java:3322) > at org.apache.hadoop.hbase.regionserver.HRegion. > batchMutate(HRegion.java:2881) > at org.apache.hadoop.hbase.regionserver.HRegion. > batchMutate(HRegion.java:2823) > at org.apache.hadoop.hbase.regionserver.RSRpcServices. > doBatchOp(RSRpcServices.java:758) > at org.apache.hadoop.hbase.regionserver.RSRpcServices. > doNonAtomicRegionMutation(RSRpcServices.java:720) > at org.apache.hadoop.hbase.regionserver.RSRpcServices. > multi(RSRpcServices.java:2168) > at org.apache.hadoop.hbase.protobuf.generated. > ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:33656) > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2188) > at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112) > at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop( > RpcExecutor.java:133) > at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108) > at java.lang.Thread.run(Thread.java:748) > > 2017-11-06 18:02:06,493 WARN [hconnection-0x52dee0d4-shared--pool423-t2] > client.AsyncProcess: #394, table=TEST_TABLE_LOCAL, attempt=11/11 > failed=40ops, last exception: org.apache.hadoop.hbase.RegionTooBusyException: > org.apache.hadoop.hbase.RegionTooBusyException: failed to get a lock in > 60000 ms. regionName=TEST_TABLE_LOCAL,\x01,1509961563238. > 9070a4f2ace4516f985c6c9e31427eee., server=ht06,16020,1509939923675 > at org.apache.hadoop.hbase.regionserver.HRegion.lock(HRegion.java:8176) > at org.apache.hadoop.hbase.regionserver.HRegion.lock(HRegion.java:8162) > at org.apache.hadoop.hbase.regionserver.HRegion. > startRegionOperation(HRegion.java:8071) > at org.apache.hadoop.hbase.regionserver.HRegion. > batchMutate(HRegion.java:2866) > at org.apache.hadoop.hbase.regionserver.HRegion. > batchMutate(HRegion.java:2823) > at org.apache.hadoop.hbase.regionserver.RSRpcServices. > doBatchOp(RSRpcServices.java:758) > at org.apache.hadoop.hbase.regionserver.RSRpcServices. > doNonAtomicRegionMutation(RSRpcServices.java:720) > at org.apache.hadoop.hbase.regionserver.RSRpcServices. > multi(RSRpcServices.java:2168) > at org.apache.hadoop.hbase.client.MultiServerCallable. > call(MultiServerCallable.java:128) > > > *The region will go back to normal after above exception,then table's > local index turn disable.* > > ------------------ Original ------------------ > *From: * "Ted Yu";<yuzhih...@gmail.com>; > *Send time:* Saturday, Nov 4, 2017 0:09 AM > *To:* "user@phoenix.apache.org"<user@phoenix.apache.org>; > *Subject: * Re: local index turn disable when region split > > Can you give us more information ? > > Release of hbase and Phoenix you use > > bq. The local index turn disable > > Can you pastebin the related exception(s) ? > > Snippet from region server log would also help. > > On Thu, Nov 2, 2017 at 11:31 PM, vergil <vergilc...@foxmail.com> wrote: > >> Hi,all: >> >> >> Here is my test table. >> *create table test_table_local(id varchar primary key,f1 varchar,f2 >> varchar) salt_buckets=3;* >> Add local index on it at the first time. >> *create local index test_index_local on test_table_local(f1);* >> >> Then upsert data into it. >> As the data increase,the region will split. >> The local index turn disable when the region splits. >> Local index data do not increase and it do not work. >> >> Here is my configuration on each master and regionserver. >> <property> <name>hbase.regionserver.wal.codec</name> >> <value>org.apache.hadoop.hbase.regionserver.wal.IndexedWALEditCodec</value> >> </property> <property> >> <name>hbase.region.server.rpc.scheduler.factory.class</name> >> <value>org.apache.hadoop.hbase.ipc.PhoenixRpcSchedulerFactory</value> >> </property> <property> <name>hbase.rpc.controllerfactory.class</name> >> <value>org.apache.hadoop.hbase.ipc.controller.ServerRpcControllerFactory</value> >> </property> >> >> I need your help,thx! >> ------------------ >> The harder, more fortunate >> >> > >
