Hi ,
One of my flink applications needs to get and put records from HBASE for
every event while processing in real time . When there are less events the
application process without any issues. when the number of events
increases we start hitting with the below mentioned exception .Can these
exceptions bring down the throughput and start to build lag . What are the
parameters we can tune at HBASE /flink side to overcome this exception . We
are seeing 7000/sec hits as minimum hits to HBase when load is normal. The
hbase table 3 region server
org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException:
Failed 1 action: IOException: 1 time,
at
org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.makeException(AsyncProcess.java:258)
at
org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.access$2000(AsyncProcess.java:238)
at
org.apache.hadoop.hbase.client.AsyncProcess.waitForAllPreviousOpsAndReset(AsyncProcess.java:1817)
at
org.apache.hadoop.hbase.client.BufferedMutatorImpl.backgroundFlushCommits(BufferedMutatorImpl.java:240)
at
org.apache.hadoop.hbase.client.BufferedMutatorImpl.flush(BufferedMutatorImpl.java:190)
at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1434)
at org.apache.hadoop.hbase.client.HTable.put(HTable.java:1018)
at org.xxxx.xxxx.xxxx.xxxx.xxxxxxxxxxxx.xxxxxx(xxxxxxx.java:202)
at
org.xxxx.xxxx.xxxx.xxxx.xxxxxxxxxxxx.xxxxxxxxxxx.xxxxxxx(xxxxxxx.java:144)
at
org.xxxx.xxxx.xxxx.xxxx.xxxxxxxxxxxx.xxxxxxxxxxx.xxxxxxxx(xxxxxxxxx.java:30)
at
org.apache.flink.streaming.api.operators.KeyedProcessOperator.processElement(KeyedProcessOperator.java:85)
at
org.apache.flink.streaming.runtime.io.StreamInputProcessor.processInput(StreamInputProcessor.java:202)
at
org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run(OneInputStreamTask.java:105)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:300)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:711)
at java.lang.Thread.run(Thread.java:745)
