Sorry, guys wrong list. Cheers Pedro
On Mon, Mar 21, 2016 at 11:00 AM, Ted Yu <[email protected]> wrote: > Have you posted this question on Phoenix mailing list ? > > Looks like you may get better answer there since the exception is related > to Phoenix coprocessor. > > Thanks > > On Mon, Mar 21, 2016 at 3:51 AM, Pedro Gandola <[email protected]> > wrote: > > > Hi, > > > > I'm using *Phoenix4.6* and in my use case I have a table that keeps a > > sliding window of 7 days worth of data. I have 3 local indexes on this > > table and in out use case we have aprox: 150 producers that are inserting > > data (in batches of 300-1500 events) in real-time. > > > > Some days ago I started to get a lot of errors like the below ones. The > > number of errors was so large that the cluster performance dropped a lot > > and my disks read bandwidth was crazy high but the write bandwidth was > > normal. I can ensure that during that period no readers were running only > > producers. > > > > ERROR [B.defaultRpcServer.handler=25,queue=5,port=16020] > > > parallel.BaseTaskRunner: Found a failed task because: > > > org.apache.hadoop.hbase.DoNotRetryIOException: *ERROR 2008 (INT10): > ERROR > > > 2008 (INT10): Unable to find cached index metadata.* > > > key=4276342695061435086 > > > > > > region=BIDDING_EVENTS,\xFEK\x17\xE4\xB1~K\x08,1458435680333.ee29454d68f5b679a8e8cc775dd0edfa. > > > *Index update failed* > > > java.util.concurrent.ExecutionException: > > > org.apache.hadoop.hbase.DoNotRetryIOException: ERROR 2008 (INT10): > ERROR > > > 2008 (INT10): Unable to find cached index metadata. > > > key=4276342695061435086 > > > > > > region=BIDDING_EVENTS,\xFEK\x17\xE4\xB1~K\x08,1458435680333.ee29454d68f5b679a8e8cc775dd0edfa. > > > Index update failed > > > Caused by: org.apache.hadoop.hbase.DoNotRetryIOException: ERROR 2008 > > > (INT10): ERROR 2008 (INT10): Unable to find cached index metadata. > > > key=4276342695061435086 > > > > > > region=BIDDING_EVENTS,\xFEK\x17\xE4\xB1~K\x08,1458435680333.ee29454d68f5b679a8e8cc775dd0edfa. > > > Index update failed > > > Caused by: java.sql.SQLException: ERROR 2008 (INT10): Unable to find > > > cached index metadata. key=4276342695061435086 > > > > > > region=BIDDING_EVENTS,\xFEK\x17\xE4\xB1~K\x08,1458435680333.ee29454d68f5b679a8e8cc775dd0edfa. > > > INFO [B.defaultRpcServer.handler=25,queue=5,port=16020] > > > parallel.TaskBatch: Aborting batch of tasks because Found a failed task > > > because: org.apache.hadoop.hbase.DoNotRetryIOException: ERROR 2008 > > (INT10): > > > ERROR 2008 (INT10): Unable to find cached index metadata. > > > key=4276342695061435086 > > > > > > region=BIDDING_EVENTS,\xFEK\x17\xE4\xB1~K\x08,1458435680333.ee29454d68f5b679a8e8cc775dd0edfa. > > > Index update failed > > > ERROR [B.defaultRpcServer.handler=25,queue=5,port=16020] > > *builder.IndexBuildManager: > > > Found a failed index update!* > > > INFO [B.defaultRpcServer.handler=25,queue=5,port=16020] > > > util.IndexManagementUtil: Rethrowing > > > org.apache.hadoop.hbase.DoNotRetryIOException: ERROR 2008 (INT10): > ERROR > > > 2008 (INT10): Unable to find cached index metadata. > > > key=4276342695061435086 > > > > > > region=BIDDING_EVENTS,\xFEK\x17\xE4\xB1~K\x08,1458435680333.ee29454d68f5b679a8e8cc775dd0edfa. > > > Index update failed > > > > > > I searched for the error and I made the following changes on the server > > side: > > > > - *phoenix.coprocessor.maxServerCacheTimeToLiveMs *from 30s to 2min > > - *phoenix.coprocessor.maxMetaDataCacheSize* from 20MB to 40MB > > > > After I changed these properties I restarted the cluster and the errors > > were gone but disks read bandwidth was still very high and I was getting > > *responseTooSlow* warnings. As a quick solution I created fresh tables > and > > then the problems were gone. > > > > Now, after one day running with new tables I started to see the problem > > again but I think this was during a major compaction but I would like to > > understand more the reasons&consequences of these problems. > > > > - What are the major consequences of these errors? I assume that index > data > > is not written within the index table, right? Then, why was the read > > bandwidth of my disks so high even without readers and after changed > those > > properties? > > > > - Is there any optimal or recommended value for the above properties or > am > > I missing some tunning on other properties for the metadata cache? > > > > Thank you, > > Pedro > > >
