Hi Ragini, Since this exception is coming from the Hbase client, I assume the issue has nothing to do with Flink directly. I would recommend carefully studying the HBase client configuration parameters, maybe setup a simple Java application that "hammers" data into Hbase at a maximum rate to understand the impact of different combinations of configuration parameters.
Best, Robert On Tue, May 4, 2021 at 5:05 AM Ragini Manjaiah <ragini.manja...@gmail.com> wrote: > Hi , > One of my flink applications needs to get and put records from HBASE for > every event while processing in real time . When there are less events the > application process without any issues. when the number of events > increases we start hitting with the below mentioned exception .Can these > exceptions bring down the throughput and start to build lag . What are the > parameters we can tune at HBASE /flink side to overcome this exception . We > are seeing 7000/sec hits as minimum hits to HBase when load is normal. The > hbase table 3 region server > > > org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 1 > action: IOException: 1 time, > at > org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.makeException(AsyncProcess.java:258) > at > org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.access$2000(AsyncProcess.java:238) > at > org.apache.hadoop.hbase.client.AsyncProcess.waitForAllPreviousOpsAndReset(AsyncProcess.java:1817) > at > org.apache.hadoop.hbase.client.BufferedMutatorImpl.backgroundFlushCommits(BufferedMutatorImpl.java:240) > at > org.apache.hadoop.hbase.client.BufferedMutatorImpl.flush(BufferedMutatorImpl.java:190) > at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1434) > at org.apache.hadoop.hbase.client.HTable.put(HTable.java:1018) > at org.xxxx.xxxx.xxxx.xxxx.xxxxxxxxxxxx.xxxxxx(xxxxxxx.java:202) > at > org.xxxx.xxxx.xxxx.xxxx.xxxxxxxxxxxx.xxxxxxxxxxx.xxxxxxx(xxxxxxx.java:144) > at > org.xxxx.xxxx.xxxx.xxxx.xxxxxxxxxxxx.xxxxxxxxxxx.xxxxxxxx(xxxxxxxxx.java:30) > at > org.apache.flink.streaming.api.operators.KeyedProcessOperator.processElement(KeyedProcessOperator.java:85) > at > org.apache.flink.streaming.runtime.io.StreamInputProcessor.processInput(StreamInputProcessor.java:202) > at > org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run(OneInputStreamTask.java:105) > at > org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:300) > at org.apache.flink.runtime.taskmanager.Task.run(Task.java:711) > at java.lang.Thread.run(Thread.java:745) > >
