Hi
According to the exception
Caused by: java.util.ConcurrentModificationException
at
java.base/java.util.ArrayList$Itr.checkForComodification(ArrayList.java:1043)
at java.base/java.util.ArrayList$Itr.next(ArrayList.java:997)
at
org.apache.hadoop.hbase.client.BufferedMutatorImpl.mutate(BufferedMutatorImpl.java:189)
at
com.flink.batch.lbs.running.hbase.udf.BatchRunningTimesHBaseUdfRichSinkFunction.flush(BatchRunningTimesHBaseUdfRichSinkFunction.java:244)
at
com.flink.batch.lbs.running.hbase.udf.BatchRunningTimesHBaseUdfRichSinkFunction.lambda$open$0(BatchRunningTimesHBaseUdfRichSinkFunction.java:97)
You can check whether there are multiple threads use the same client
Best,
Shammon
On Fri, Feb 24, 2023 at 4:36 PM Weihua Hu <huweihua....@gmail.com> wrote:
> Hi,
>
> The cause of the error is clearly shown in the log. You need to check the
> logic in 'com.flink.batch.lbs.running.hbase.udf.
> BatchRunningTimesHBaseUdfRichSinkFunction#flush'
>
>
> Best,
> Weihua
>
>
> On Fri, Feb 24, 2023 at 4:26 PM simple <1028108...@qq.com> wrote:
>
>>
>> 2023-02-24 16:05:26,104 INFO
>> org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source:
>> Custom Source (1/1)
>> (9644f5bc58ed204c6d31bb133156fa95_bc764cd8ddf7a0cff126f51c16239658_0_0)
>> switched from RUNNING to CANCELING.
>> 2023-02-24 16:05:26,108 INFO
>> org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Flat Map (
>> 1/2)
>> (9644f5bc58ed204c6d31bb133156fa95_0a448493b4782967b150582570326227_0_0)
>> switched from RUNNING to CANCELING.
>> 2023-02-24 16:05:26,108 INFO
>> org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Flat Map (
>> 2/2)
>> (9644f5bc58ed204c6d31bb133156fa95_0a448493b4782967b150582570326227_1_0)
>> switched from RUNNING to CANCELING.
>> 2023-02-24 16:05:26,108 INFO
>> org.apache.flink.runtime.executiongraph.ExecutionGraph [] -
>> KeyedProcess -> Sink: Unnamed (2/2)
>> (9644f5bc58ed204c6d31bb133156fa95_e70bbd798b564e0a50e10e343f1ac56b_1_0)
>> switched from RUNNING to CANCELING.
>> 2023-02-24 16:05:26,137 INFO
>> org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Flat Map (
>> 1/2)
>> (9644f5bc58ed204c6d31bb133156fa95_0a448493b4782967b150582570326227_0_0)
>> switched from CANCELING to CANCELED.
>> 2023-02-24 16:05:26,140 INFO
>> org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Flat Map (
>> 2/2)
>> (9644f5bc58ed204c6d31bb133156fa95_0a448493b4782967b150582570326227_1_0)
>> switched from CANCELING to CANCELED.
>> 2023-02-24 16:05:26,141 INFO
>> org.apache.flink.runtime.executiongraph.ExecutionGraph [] -
>> KeyedProcess -> Sink: Unnamed (2/2)
>> (9644f5bc58ed204c6d31bb133156fa95_e70bbd798b564e0a50e10e343f1ac56b_1_0)
>> switched from CANCELING to CANCELED.
>> 2023-02-24 16:05:26,145 INFO
>> org.apache.flink.runtime.resourcemanager.slotmanager.DeclarativeSlotManager
>> [] - Received resource requirements from job
>> 299ff4a6ea193d7c110a19a3de33edab:
>> [ResourceRequirement{resourceProfile=ResourceProfile{UNKNOWN},
>> numberOfRequiredSlots=1}]
>> 2023-02-24 16:05:26,152 INFO
>> org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source:
>> Custom Source (1/1)
>> (9644f5bc58ed204c6d31bb133156fa95_bc764cd8ddf7a0cff126f51c16239658_0_0)
>> switched from CANCELING to CANCELED.
>> 2023-02-24 16:05:26,155 INFO
>> org.apache.flink.runtime.resourcemanager.slotmanager.DeclarativeSlotManager
>> [] - Clearing resource requirements of job 299ff4a6ea193d7c110a19a3de33edab
>> 2023-02-24 16:05:26,154 INFO
>> org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Job [
>> 10204-08]BatchRunningTimesLBSHBaseJobApp
>> (299ff4a6ea193d7c110a19a3de33edab) switched from state FAILING to FAILED.
>> org.apache.flink.runtime.JobException: Recovery is suppressed by
>> NoRestartBackoffTimeStrategy
>> at
>> org.apache.flink.runtime.executiongraph.failover.flip1.ExecutionFailureHandler.handleFailure(ExecutionFailureHandler.java:139)
>> ~[flink-runtime-1.16.0.jar:1.16.0]
>> at
>> org.apache.flink.runtime.executiongraph.failover.flip1.ExecutionFailureHandler.getFailureHandlingResult(ExecutionFailureHandler.java:83)
>> ~[flink-runtime-1.16.0.jar:1.16.0]
>> at
>> org.apache.flink.runtime.scheduler.DefaultScheduler.recordTaskFailure(DefaultScheduler.java:256)
>> ~[flink-runtime-1.16.0.jar:1.16.0]
>> at
>> org.apache.flink.runtime.scheduler.DefaultScheduler.handleTaskFailure(DefaultScheduler.java:247)
>> ~[flink-runtime-1.16.0.jar:1.16.0]
>> at
>> org.apache.flink.runtime.scheduler.DefaultScheduler.onTaskFailed(DefaultScheduler.java:240)
>> ~[flink-runtime-1.16.0.jar:1.16.0]
>> at
>> org.apache.flink.runtime.scheduler.SchedulerBase.onTaskExecutionStateUpdate(SchedulerBase.java:738)
>> ~[flink-runtime-1.16.0.jar:1.16.0]
>> at
>> org.apache.flink.runtime.scheduler.SchedulerBase.updateTaskExecutionState(SchedulerBase.java:715)
>> ~[flink-runtime-1.16.0.jar:1.16.0]
>> at
>> org.apache.flink.runtime.scheduler.SchedulerNG.updateTaskExecutionState(SchedulerNG.java:78)
>> ~[flink-runtime-1.16.0.jar:1.16.0]
>> at
>> org.apache.flink.runtime.jobmaster.JobMaster.updateTaskExecutionState(JobMaster.java:477)
>> ~[flink-runtime-1.16.0.jar:1.16.0]
>> at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> ~[?:?]
>> at
>> jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>> ~[?:?]
>> at
>> jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> ~[?:?]
>> at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
>> at
>> org.apache.flink.runtime.rpc.akka.AkkaRpcActor.lambda$handleRpcInvocation$1(AkkaRpcActor.java:309)
>> ~[flink-rpc-akka_24401257-5903-4217-89f4-09e8c9e6f005.jar:1.16.0]
>> at
>> org.apache.flink.runtime.concurrent.akka.ClassLoadingUtils.runWithContextClassLoader(ClassLoadingUtils.java:83)
>> ~[flink-rpc-akka_24401257-5903-4217-89f4-09e8c9e6f005.jar:1.16.0]
>> at
>> org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcInvocation(AkkaRpcActor.java:307)
>> ~[flink-rpc-akka_24401257-5903-4217-89f4-09e8c9e6f005.jar:1.16.0]
>> at
>> org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:222)
>> ~[flink-rpc-akka_24401257-5903-4217-89f4-09e8c9e6f005.jar:1.16.0]
>> at
>> org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:84)
>> ~[flink-rpc-akka_24401257-5903-4217-89f4-09e8c9e6f005.jar:1.16.0]
>> at
>> org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:168)
>> ~[flink-rpc-akka_24401257-5903-4217-89f4-09e8c9e6f005.jar:1.16.0]
>> at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:24)
>> [flink-rpc-akka_24401257-5903-4217-89f4-09e8c9e6f005.jar:1.16.0]
>> at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:20)
>> [flink-rpc-akka_24401257-5903-4217-89f4-09e8c9e6f005.jar:1.16.0]
>> at scala.PartialFunction.applyOrElse(PartialFunction.scala:123)
>> [flink-rpc-akka_24401257-5903-4217-89f4-09e8c9e6f005.jar:1.16.0]
>> at scala.PartialFunction.applyOrElse$(PartialFunction.scala:122)
>> [flink-rpc-akka_24401257-5903-4217-89f4-09e8c9e6f005.jar:1.16.0]
>> at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:20)
>> [flink-rpc-akka_24401257-5903-4217-89f4-09e8c9e6f005.jar:1.16.0]
>> at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
>> [flink-rpc-akka_24401257-5903-4217-89f4-09e8c9e6f005.jar:1.16.0]
>> at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172)
>> [flink-rpc-akka_24401257-5903-4217-89f4-09e8c9e6f005.jar:1.16.0]
>> at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172)
>> [flink-rpc-akka_24401257-5903-4217-89f4-09e8c9e6f005.jar:1.16.0]
>> at akka.actor.Actor.aroundReceive(Actor.scala:537)
>> [flink-rpc-akka_24401257-5903-4217-89f4-09e8c9e6f005.jar:1.16.0]
>> at akka.actor.Actor.aroundReceive$(Actor.scala:535)
>> [flink-rpc-akka_24401257-5903-4217-89f4-09e8c9e6f005.jar:1.16.0]
>> at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:220)
>> [flink-rpc-akka_24401257-5903-4217-89f4-09e8c9e6f005.jar:1.16.0]
>> at akka.actor.ActorCell.receiveMessage(ActorCell.scala:580)
>> [flink-rpc-akka_24401257-5903-4217-89f4-09e8c9e6f005.jar:1.16.0]
>> at akka.actor.ActorCell.invoke(ActorCell.scala:548)
>> [flink-rpc-akka_24401257-5903-4217-89f4-09e8c9e6f005.jar:1.16.0]
>> at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:270)
>> [flink-rpc-akka_24401257-5903-4217-89f4-09e8c9e6f005.jar:1.16.0]
>> at akka.dispatch.Mailbox.run(Mailbox.scala:231)
>> [flink-rpc-akka_24401257-5903-4217-89f4-09e8c9e6f005.jar:1.16.0]
>> at akka.dispatch.Mailbox.exec(Mailbox.scala:243)
>> [flink-rpc-akka_24401257-5903-4217-89f4-09e8c9e6f005.jar:1.16.0]
>> at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290) [?:?]
>> at
>> java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020)
>> [?:?]
>> at java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656) [?:?]
>> at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594)
>> [?:?]
>> at
>> java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183)
>> [?:?]
>> Caused by: org.apache.flink.runtime.taskmanager.AsynchronousException:
>> Caught exception while processing timer.
>> at
>> org.apache.flink.streaming.runtime.tasks.StreamTask$StreamTaskAsyncExceptionHandler.handleAsyncException(StreamTask.java:1568)
>> ~[flink-streaming-java-1.16.0.jar:1.16.0]
>> at
>> org.apache.flink.streaming.runtime.tasks.StreamTask.handleAsyncException(StreamTask.java:1543)
>> ~[flink-streaming-java-1.16.0.jar:1.16.0]
>> at
>> org.apache.flink.streaming.runtime.tasks.StreamTask.invokeProcessingTimeCallback(StreamTask.java:1683)
>> ~[flink-streaming-java-1.16.0.jar:1.16.0]
>> at
>> org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$null$22(StreamTask.java:1672)
>> ~[flink-streaming-java-1.16.0.jar:1.16.0]
>> at
>> org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.runThrowing(StreamTaskActionExecutor.java:50)
>> ~[flink-streaming-java-1.16.0.jar:1.16.0]
>> at
>> org.apache.flink.streaming.runtime.tasks.mailbox.Mail.run(Mail.java:90)
>> ~[flink-streaming-java-1.16.0.jar:1.16.0]
>> at
>> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMail(MailboxProcessor.java:398)
>> ~[flink-streaming-java-1.16.0.jar:1.16.0]
>> at
>> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMailsNonBlocking(MailboxProcessor.java:383)
>> ~[flink-streaming-java-1.16.0.jar:1.16.0]
>> at
>> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMail(MailboxProcessor.java:345)
>> ~[flink-streaming-java-1.16.0.jar:1.16.0]
>> at
>> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:229)
>> ~[flink-streaming-java-1.16.0.jar:1.16.0]
>> at
>> org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:831)
>> ~[flink-streaming-java-1.16.0.jar:1.16.0]
>> at
>> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:780)
>> ~[flink-streaming-java-1.16.0.jar:1.16.0]
>> at
>> org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:935)
>> ~[flink-runtime-1.16.0.jar:1.16.0]
>> at
>> org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:914)
>> ~[flink-runtime-1.16.0.jar:1.16.0]
>> at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:728)
>> ~[flink-runtime-1.16.0.jar:1.16.0]
>> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:550)
>> ~[flink-runtime-1.16.0.jar:1.16.0]
>> at java.lang.Thread.run(Thread.java:834) ~[?:?]
>> Caused by: org.apache.flink.streaming.runtime.tasks.TimerException:
>> org.apache.flink.streaming.runtime.tasks.ExceptionInChainedOperatorException:
>> Could not forward element to next operator
>> at
>> org.apache.flink.streaming.runtime.tasks.StreamTask.invokeProcessingTimeCallback(StreamTask.java:1683)
>> ~[flink-streaming-java-1.16.0.jar:1.16.0]
>> at
>> org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$null$22(StreamTask.java:1672)
>> ~[flink-streaming-java-1.16.0.jar:1.16.0]
>> at
>> org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.runThrowing(StreamTaskActionExecutor.java:50)
>> ~[flink-streaming-java-1.16.0.jar:1.16.0]
>> at
>> org.apache.flink.streaming.runtime.tasks.mailbox.Mail.run(Mail.java:90)
>> ~[flink-streaming-java-1.16.0.jar:1.16.0]
>> at
>> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMail(MailboxProcessor.java:398)
>> ~[flink-streaming-java-1.16.0.jar:1.16.0]
>> at
>> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMailsNonBlocking(MailboxProcessor.java:383)
>> ~[flink-streaming-java-1.16.0.jar:1.16.0]
>> at
>> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMail(MailboxProcessor.java:345)
>> ~[flink-streaming-java-1.16.0.jar:1.16.0]
>> at
>> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:229)
>> ~[flink-streaming-java-1.16.0.jar:1.16.0]
>> at
>> org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:831)
>> ~[flink-streaming-java-1.16.0.jar:1.16.0]
>> at
>> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:780)
>> ~[flink-streaming-java-1.16.0.jar:1.16.0]
>> at
>> org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:935)
>> ~[flink-runtime-1.16.0.jar:1.16.0]
>> at
>> org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:914)
>> ~[flink-runtime-1.16.0.jar:1.16.0]
>> at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:728)
>> ~[flink-runtime-1.16.0.jar:1.16.0]
>> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:550)
>> ~[flink-runtime-1.16.0.jar:1.16.0]
>> at java.lang.Thread.run(Thread.java:834) ~[?:?]
>> Caused by:
>> org.apache.flink.streaming.runtime.tasks.ExceptionInChainedOperatorException:
>> Could not forward element to next operator
>> at
>> org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.pushToOperator(CopyingChainingOutput.java:99)
>> ~[flink-streaming-java-1.16.0.jar:1.16.0]
>> at
>> org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:57)
>> ~[flink-streaming-java-1.16.0.jar:1.16.0]
>> at
>> org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:29)
>> ~[flink-streaming-java-1.16.0.jar:1.16.0]
>> at
>> org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:56)
>> ~[flink-streaming-java-1.16.0.jar:1.16.0]
>> at
>> org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:29)
>> ~[flink-streaming-java-1.16.0.jar:1.16.0]
>> at
>> org.apache.flink.streaming.api.operators.TimestampedCollector.collect(TimestampedCollector.java:51)
>> ~[flink-streaming-java-1.16.0.jar:1.16.0]
>> at
>> com.flink.batch.lbs.running.hbase.udf.BatchRunningTimesHBaseUdfKeyedProcessFunctionV2.transformation(BatchRunningTimesHBaseUdfKeyedProcessFunctionV2.java:87)
>> ~[?:?]
>> at
>> com.flink.batch.lbs.running.hbase.udf.BatchRunningTimesHBaseUdfKeyedProcessFunctionV2.resultOut(BatchRunningTimesHBaseUdfKeyedProcessFunctionV2.java:76)
>> ~[?:?]
>> at
>> com.flink.batch.lbs.running.hbase.udf.BatchRunningTimesHBaseUdfKeyedProcessFunctionV2.onTimerCallback(BatchRunningTimesHBaseUdfKeyedProcessFunctionV2.java:48)
>> ~[?:?]
>> at
>> com.flink.batch.lbs.running.hbase.udf.BatchRunningTimesHBaseUdfKeyedProcessFunctionV2.onTimer(BatchRunningTimesHBaseUdfKeyedProcessFunctionV2.java:43)
>> ~[?:?]
>> at
>> org.apache.flink.streaming.api.operators.KeyedProcessOperator.invokeUserFunction(KeyedProcessOperator.java:91)
>> ~[flink-streaming-java-1.16.0.jar:1.16.0]
>> at
>> org.apache.flink.streaming.api.operators.KeyedProcessOperator.onProcessingTime(KeyedProcessOperator.java:76)
>> ~[flink-streaming-java-1.16.0.jar:1.16.0]
>> at
>> org.apache.flink.streaming.api.operators.InternalTimerServiceImpl.onProcessingTime(InternalTimerServiceImpl.java:284)
>> ~[flink-streaming-java-1.16.0.jar:1.16.0]
>> at
>> org.apache.flink.streaming.runtime.tasks.StreamTask.invokeProcessingTimeCallback(StreamTask.java:1681)
>> ~[flink-streaming-java-1.16.0.jar:1.16.0]
>> at
>> org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$null$22(StreamTask.java:1672)
>> ~[flink-streaming-java-1.16.0.jar:1.16.0]
>> at
>> org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.runThrowing(StreamTaskActionExecutor.java:50)
>> ~[flink-streaming-java-1.16.0.jar:1.16.0]
>> at
>> org.apache.flink.streaming.runtime.tasks.mailbox.Mail.run(Mail.java:90)
>> ~[flink-streaming-java-1.16.0.jar:1.16.0]
>> at
>> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMail(MailboxProcessor.java:398)
>> ~[flink-streaming-java-1.16.0.jar:1.16.0]
>> at
>> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMailsNonBlocking(MailboxProcessor.java:383)
>> ~[flink-streaming-java-1.16.0.jar:1.16.0]
>> at
>> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMail(MailboxProcessor.java:345)
>> ~[flink-streaming-java-1.16.0.jar:1.16.0]
>> at
>> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:229)
>> ~[flink-streaming-java-1.16.0.jar:1.16.0]
>> at
>> org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:831)
>> ~[flink-streaming-java-1.16.0.jar:1.16.0]
>> at
>> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:780)
>> ~[flink-streaming-java-1.16.0.jar:1.16.0]
>> at
>> org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:935)
>> ~[flink-runtime-1.16.0.jar:1.16.0]
>> at
>> org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:914)
>> ~[flink-runtime-1.16.0.jar:1.16.0]
>> at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:728)
>> ~[flink-runtime-1.16.0.jar:1.16.0]
>> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:550)
>> ~[flink-runtime-1.16.0.jar:1.16.0]
>> at java.lang.Thread.run(Thread.java:834) ~[?:?]
>> Caused by: java.lang.RuntimeException: exception
>> at
>> com.flink.batch.lbs.running.hbase.udf.BatchRunningTimesHBaseUdfRichSinkFunction.checkErrorAndRethrow(BatchRunningTimesHBaseUdfRichSinkFunction.java:264)
>> ~[?:?]
>> at
>> com.flink.batch.lbs.running.hbase.udf.BatchRunningTimesHBaseUdfRichSinkFunction.flush(BatchRunningTimesHBaseUdfRichSinkFunction.java:257)
>> ~[?:?]
>> at
>> com.flink.batch.lbs.running.hbase.udf.BatchRunningTimesHBaseUdfRichSinkFunction.invoke(BatchRunningTimesHBaseUdfRichSinkFunction.java:235)
>> ~[?:?]
>> at
>> com.flink.batch.lbs.running.hbase.udf.BatchRunningTimesHBaseUdfRichSinkFunction.invoke(BatchRunningTimesHBaseUdfRichSinkFunction.java:37)
>> ~[?:?]
>> at
>> org.apache.flink.streaming.api.operators.StreamSink.processElement(StreamSink.java:54)
>> ~[flink-streaming-java-1.16.0.jar:1.16.0]
>> at
>> org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.pushToOperator(CopyingChainingOutput.java:82)
>> ~[flink-streaming-java-1.16.0.jar:1.16.0]
>> at
>> org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:57)
>> ~[flink-streaming-java-1.16.0.jar:1.16.0]
>> at
>> org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:29)
>> ~[flink-streaming-java-1.16.0.jar:1.16.0]
>> at
>> org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:56)
>> ~[flink-streaming-java-1.16.0.jar:1.16.0]
>> at
>> org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:29)
>> ~[flink-streaming-java-1.16.0.jar:1.16.0]
>> at
>> org.apache.flink.streaming.api.operators.TimestampedCollector.collect(TimestampedCollector.java:51)
>> ~[flink-streaming-java-1.16.0.jar:1.16.0]
>> at
>> com.flink.batch.lbs.running.hbase.udf.BatchRunningTimesHBaseUdfKeyedProcessFunctionV2.transformation(BatchRunningTimesHBaseUdfKeyedProcessFunctionV2.java:87)
>> ~[?:?]
>> at
>> com.flink.batch.lbs.running.hbase.udf.BatchRunningTimesHBaseUdfKeyedProcessFunctionV2.resultOut(BatchRunningTimesHBaseUdfKeyedProcessFunctionV2.java:76)
>> ~[?:?]
>> at
>> com.flink.batch.lbs.running.hbase.udf.BatchRunningTimesHBaseUdfKeyedProcessFunctionV2.onTimerCallback(BatchRunningTimesHBaseUdfKeyedProcessFunctionV2.java:48)
>> ~[?:?]
>> at
>> com.flink.batch.lbs.running.hbase.udf.BatchRunningTimesHBaseUdfKeyedProcessFunctionV2.onTimer(BatchRunningTimesHBaseUdfKeyedProcessFunctionV2.java:43)
>> ~[?:?]
>> at
>> org.apache.flink.streaming.api.operators.KeyedProcessOperator.invokeUserFunction(KeyedProcessOperator.java:91)
>> ~[flink-streaming-java-1.16.0.jar:1.16.0]
>> at
>> org.apache.flink.streaming.api.operators.KeyedProcessOperator.onProcessingTime(KeyedProcessOperator.java:76)
>> ~[flink-streaming-java-1.16.0.jar:1.16.0]
>> at
>> org.apache.flink.streaming.api.operators.InternalTimerServiceImpl.onProcessingTime(InternalTimerServiceImpl.java:284)
>> ~[flink-streaming-java-1.16.0.jar:1.16.0]
>> at
>> org.apache.flink.streaming.runtime.tasks.StreamTask.invokeProcessingTimeCallback(StreamTask.java:1681)
>> ~[flink-streaming-java-1.16.0.jar:1.16.0]
>> at
>> org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$null$22(StreamTask.java:1672)
>> ~[flink-streaming-java-1.16.0.jar:1.16.0]
>> at
>> org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.runThrowing(StreamTaskActionExecutor.java:50)
>> ~[flink-streaming-java-1.16.0.jar:1.16.0]
>> at
>> org.apache.flink.streaming.runtime.tasks.mailbox.Mail.run(Mail.java:90)
>> ~[flink-streaming-java-1.16.0.jar:1.16.0]
>> at
>> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMail(MailboxProcessor.java:398)
>> ~[flink-streaming-java-1.16.0.jar:1.16.0]
>> at
>> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMailsNonBlocking(MailboxProcessor.java:383)
>> ~[flink-streaming-java-1.16.0.jar:1.16.0]
>> at
>> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMail(MailboxProcessor.java:345)
>> ~[flink-streaming-java-1.16.0.jar:1.16.0]
>> at
>> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:229)
>> ~[flink-streaming-java-1.16.0.jar:1.16.0]
>> at
>> org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:831)
>> ~[flink-streaming-java-1.16.0.jar:1.16.0]
>> at
>> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:780)
>> ~[flink-streaming-java-1.16.0.jar:1.16.0]
>> at
>> org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:935)
>> ~[flink-runtime-1.16.0.jar:1.16.0]
>> at
>> org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:914)
>> ~[flink-runtime-1.16.0.jar:1.16.0]
>> at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:728)
>> ~[flink-runtime-1.16.0.jar:1.16.0]
>> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:550)
>> ~[flink-runtime-1.16.0.jar:1.16.0]
>> at java.lang.Thread.run(Thread.java:834) ~[?:?]
>> Caused by: java.util.ConcurrentModificationException
>> at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:1043)
>> ~[?:?]
>> at java.util.ArrayList$Itr.next(ArrayList.java:997) ~[?:?]
>> at
>> org.apache.hadoop.hbase.client.BufferedMutatorImpl.mutate(BufferedMutatorImpl.java:189)
>> ~[hbase-client-2.2.2.jar:2.2.2]
>> at
>> com.flink.batch.lbs.running.hbase.udf.BatchRunningTimesHBaseUdfRichSinkFunction.flush(BatchRunningTimesHBaseUdfRichSinkFunction.java:244)
>> ~[?:?]
>> at
>> com.flink.batch.lbs.running.hbase.udf.BatchRunningTimesHBaseUdfRichSinkFunction.lambda$open$0(BatchRunningTimesHBaseUdfRichSinkFunction.java:97)
>> ~[?:?]
>> at
>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
>> ~[?:?]
>> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) ~[?:?]
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
>> ~[?:?]
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
>> ~[?:?]
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>> ~[?:?]
>> at java.lang.Thread.run(Thread.java:834) ~[?:?]
>> 2023-02-24 16:05:26,158 INFO
>> org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Stopping
>> checkpoint coordinator for job 299ff4a6ea193d7c110a19a3de33edab.
>> 2023-02-24 16:05:26,178 INFO
>> org.apache.flink.runtime.dispatcher.StandaloneDispatcher [] - Job
>> 299ff4a6ea193d7c110a19a3de33edab reached terminal state FAILED.
>> org.apache.flink.runtime.JobException: Recovery is suppressed by
>> NoRestartBackoffTimeStrategy
>> at
>> org.apache.flink.runtime.executiongraph.failover.flip1.ExecutionFailureHandler.handleFailure(ExecutionFailureHandler.java:139)
>> at
>> org.apache.flink.runtime.executiongraph.failover.flip1.ExecutionFailureHandler.getFailureHandlingResult(ExecutionFailureHandler.java:83)
>> at
>> org.apache.flink.runtime.scheduler.DefaultScheduler.recordTaskFailure(DefaultScheduler.java:256)
>> at
>> org.apache.flink.runtime.scheduler.DefaultScheduler.handleTaskFailure(DefaultScheduler.java:247)
>> at
>> org.apache.flink.runtime.scheduler.DefaultScheduler.onTaskFailed(DefaultScheduler.java:240)
>> at
>> org.apache.flink.runtime.scheduler.SchedulerBase.onTaskExecutionStateUpdate(SchedulerBase.java:738)
>> at
>> org.apache.flink.runtime.scheduler.SchedulerBase.updateTaskExecutionState(SchedulerBase.java:715)
>> at
>> org.apache.flink.runtime.scheduler.SchedulerNG.updateTaskExecutionState(SchedulerNG.java:78)
>> at
>> org.apache.flink.runtime.jobmaster.JobMaster.updateTaskExecutionState(JobMaster.java:477)
>> at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
>> Method)
>> at
>> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62
>> )
>> at
>> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43
>> )
>> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
>> at
>> org.apache.flink.runtime.rpc.akka.AkkaRpcActor.lambda$handleRpcInvocation$1(AkkaRpcActor.java:309)
>> at
>> org.apache.flink.runtime.concurrent.akka.ClassLoadingUtils.runWithContextClassLoader(ClassLoadingUtils.java:83)
>> at
>> org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcInvocation(AkkaRpcActor.java:307)
>> at
>> org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:222)
>> at
>> org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:84)
>> at
>> org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:168)
>> at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:24)
>> at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:20)
>> at scala.PartialFunction.applyOrElse(PartialFunction.scala:123)
>> at scala.PartialFunction.applyOrElse$(PartialFunction.scala:122)
>> at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:20)
>> at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
>> at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172)
>> at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172)
>> at akka.actor.Actor.aroundReceive(Actor.scala:537)
>> at akka.actor.Actor.aroundReceive$(Actor.scala:535)
>> at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:220)
>> at akka.actor.ActorCell.receiveMessage(ActorCell.scala:580)
>> at akka.actor.ActorCell.invoke(ActorCell.scala:548)
>> at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:270)
>> at akka.dispatch.Mailbox.run(Mailbox.scala:231)
>> at akka.dispatch.Mailbox.exec(Mailbox.scala:243)
>> at
>> java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290)
>> at
>> java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020
>> )
>> at
>> java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656)
>> at
>> java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594
>> )
>> at
>> java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183
>> )
>> Caused by: org.apache.flink.runtime.taskmanager.AsynchronousException:
>> Caught exception while processing timer.
>> at
>> org.apache.flink.streaming.runtime.tasks.StreamTask$StreamTaskAsyncExceptionHandler.handleAsyncException(StreamTask.java:1568)
>> at
>> org.apache.flink.streaming.runtime.tasks.StreamTask.handleAsyncException(StreamTask.java:1543)
>> at
>> org.apache.flink.streaming.runtime.tasks.StreamTask.invokeProcessingTimeCallback(StreamTask.java:1683)
>> at
>> org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$null$22(StreamTask.java:1672)
>> at
>> org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.runThrowing(StreamTaskActionExecutor.java:50)
>> at org.apache.flink.streaming.runtime.tasks.mailbox.Mail.run(Mail.java:90)
>> at
>> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMail(MailboxProcessor.java:398)
>> at
>> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMailsNonBlocking(MailboxProcessor.java:383)
>> at
>> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMail(MailboxProcessor.java:345)
>> at
>> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:229)
>> at
>> org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:831)
>> at
>> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:780)
>> at
>> org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:935)
>> at
>> org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:914)
>> at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:728)
>> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:550)
>> at java.base/java.lang.Thread.run(Thread.java:834)
>> Caused by:
>> TimerException{org.apache.flink.streaming.runtime.tasks.ExceptionInChainedOperatorException:
>> Could not forward element to next operator}
>> ... 15 more
>> Caused by:
>> org.apache.flink.streaming.runtime.tasks.ExceptionInChainedOperatorException:
>> Could not forward element to next operator
>> at
>> org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.pushToOperator(CopyingChainingOutput.java:99)
>> at
>> org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:57)
>> at
>> org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:29)
>> at
>> org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:56)
>> at
>> org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:29)
>> at
>> org.apache.flink.streaming.api.operators.TimestampedCollector.collect(TimestampedCollector.java:51)
>> at
>> com.flink.batch.lbs.running.hbase.udf.BatchRunningTimesHBaseUdfKeyedProcessFunctionV2.transformation(BatchRunningTimesHBaseUdfKeyedProcessFunctionV2.java:87)
>> at
>> com.flink.batch.lbs.running.hbase.udf.BatchRunningTimesHBaseUdfKeyedProcessFunctionV2.resultOut(BatchRunningTimesHBaseUdfKeyedProcessFunctionV2.java:76)
>> at
>> com.flink.batch.lbs.running.hbase.udf.BatchRunningTimesHBaseUdfKeyedProcessFunctionV2.onTimerCallback(BatchRunningTimesHBaseUdfKeyedProcessFunctionV2.java:48)
>> at
>> com.flink.batch.lbs.running.hbase.udf.BatchRunningTimesHBaseUdfKeyedProcessFunctionV2.onTimer(BatchRunningTimesHBaseUdfKeyedProcessFunctionV2.java:43)
>> at
>> org.apache.flink.streaming.api.operators.KeyedProcessOperator.invokeUserFunction(KeyedProcessOperator.java:91)
>> at
>> org.apache.flink.streaming.api.operators.KeyedProcessOperator.onProcessingTime(KeyedProcessOperator.java:76)
>> at
>> org.apache.flink.streaming.api.operators.InternalTimerServiceImpl.onProcessingTime(InternalTimerServiceImpl.java:284)
>> at
>> org.apache.flink.streaming.runtime.tasks.StreamTask.invokeProcessingTimeCallback(StreamTask.java:1681)
>> ... 14 more
>> Caused by: java.lang.RuntimeException: exception
>> at
>> com.flink.batch.lbs.running.hbase.udf.BatchRunningTimesHBaseUdfRichSinkFunction.checkErrorAndRethrow(BatchRunningTimesHBaseUdfRichSinkFunction.java:264)
>> at
>> com.flink.batch.lbs.running.hbase.udf.BatchRunningTimesHBaseUdfRichSinkFunction.flush(BatchRunningTimesHBaseUdfRichSinkFunction.java:257)
>> at
>> com.flink.batch.lbs.running.hbase.udf.BatchRunningTimesHBaseUdfRichSinkFunction.invoke(BatchRunningTimesHBaseUdfRichSinkFunction.java:235)
>> at
>> com.flink.batch.lbs.running.hbase.udf.BatchRunningTimesHBaseUdfRichSinkFunction.invoke(BatchRunningTimesHBaseUdfRichSinkFunction.java:37)
>> at
>> org.apache.flink.streaming.api.operators.StreamSink.processElement(StreamSink.java:54)
>> at
>> org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.pushToOperator(CopyingChainingOutput.java:82)
>> ... 27 more
>> Caused by: java.util.ConcurrentModificationException
>> at
>> java.base/java.util.ArrayList$Itr.checkForComodification(ArrayList.java:1043
>> )
>> at java.base/java.util.ArrayList$Itr.next(ArrayList.java:997)
>> at
>> org.apache.hadoop.hbase.client.BufferedMutatorImpl.mutate(BufferedMutatorImpl.java:189)
>> at
>> com.flink.batch.lbs.running.hbase.udf.BatchRunningTimesHBaseUdfRichSinkFunction.flush(BatchRunningTimesHBaseUdfRichSinkFunction.java:244)
>> at
>> com.flink.batch.lbs.running.hbase.udf.BatchRunningTimesHBaseUdfRichSinkFunction.lambda$open$0(BatchRunningTimesHBaseUdfRichSinkFunction.java:97)
>> at
>> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515
>> )
>> at
>> java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305
>> )
>> at
>> java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305
>> )
>> at
>> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128
>> )
>> at
>> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628
>> )
>> ... 1 more
>> 2023-02-24 16:05:26,225 INFO
>> org.apache.flink.runtime.dispatcher.StandaloneDispatcher [] - Job
>> 299ff4a6ea193d7c110a19a3de33edab has been registered for cleanup in the
>> JobResultStore after reaching a terminal state.
>> 2023-02-24 16:05:26,231 INFO
>> org.apache.flink.runtime.jobmaster.JobMaster [] - Stopping
>> the JobMaster for job '[10204-08]BatchRunningTimesLBSHBaseJobApp'
>> (299ff4a6ea193d7c110a19a3de33edab).
>> 2023-02-24 16:05:26,237 INFO
>> org.apache.flink.runtime.checkpoint.StandaloneCompletedCheckpointStore [] -
>> Shutting down
>> 2023-02-24 16:05:26,239 INFO
>> org.apache.flink.runtime.jobmaster.JobMaster [] -
>> Disconnect TaskExecutor localhost:45013-1b5bd3 because: Stopping JobMaster
>> for job '[10204-08]BatchRunningTimesLBSHBaseJobApp'
>> (299ff4a6ea193d7c110a19a3de33edab).
>> 2023-02-24 16:05:26,240 INFO
>> org.apache.flink.runtime.jobmaster.slotpool.DefaultDeclarativeSlotPool [] -
>> Releasing slot [ce80357b0859df0b268566294786c8a9].
>> 2023-02-24 16:05:26,242 INFO
>> org.apache.flink.runtime.jobmaster.slotpool.DefaultDeclarativeSlotPool [] -
>> Releasing slot [7f78bde0f41f2cf0931a01a54db98bfb].
>> 2023-02-24 16:05:26,243 INFO
>> org.apache.flink.runtime.jobmaster.JobMaster [] - Close
>> ResourceManager connection 4d0adaf4e63930c3693def906b4a3e7e: Stopping
>> JobMaster for job '[10204-08]BatchRunningTimesLBSHBaseJobApp'
>> (299ff4a6ea193d7c110a19a3de33edab).
>> 2023-02-24 16:05:26,246 INFO
>> org.apache.flink.runtime.resourcemanager.StandaloneResourceManager [] -
>> Disconnect job manager 00000000000000000000000000000000
>> @akka.tcp://flink@localhost:6123/user/rpc/jobmanager_2 for job
>> 299ff4a6ea193d7c110a19a3de33edab from the resource manager.
>>
>> ------------------------------
>> 冉庆超
>> 1028108...@qq.com
>>
>> <https://wx.mail.qq.com/home/index?t=readmail_businesscard_midpage&nocheck=true&name=%E5%86%89%E5%BA%86%E8%B6%85&icon=http%3A%2F%2Fthirdqq.qlogo.cn%2Fg%3Fb%3Dsdk%26k%3DKiaSrhuQkdzkuLbXWejpF0g%26s%3D100%26t%3D1570761082%3Frand%3D1641817208&mail=1028108734%40qq.com&code=>
>>
>>
>
