WhatAKitty opened a new issue, #5157: URL: https://github.com/apache/seatunnel/issues/5157
### Search before asking - [X] I had searched in the [issues](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues. ### What happened wait checkpoint completed: 204 wait checkpoint completed: 205 ..... and so on. The bellow is some detailed msg: ``` 2023-07-26 12:56:11,300 DEBUG org.apache.seatunnel.engine.server.service.slot.DefaultSlotService - start send heartbeat to resource manager, this address: [ds01]:5801 2023-07-26 12:56:11,300 DEBUG org.apache.seatunnel.engine.server.resourcemanager.AbstractResourceManager - received worker heartbeat from: [ds01]:5801 2023-07-26 12:56:16,300 DEBUG org.apache.seatunnel.engine.server.service.slot.DefaultSlotService - start send heartbeat to resource manager, this address: [ds01]:5801 2023-07-26 12:56:16,300 DEBUG org.apache.seatunnel.engine.server.resourcemanager.AbstractResourceManager - received worker heartbeat from: [ds01]:5801 2023-07-26 12:56:18,483 INFO org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator - wait checkpoint completed: 204 2023-07-26 12:56:18,483 DEBUG org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator - trigger checkpoint barrier 736433868642451457/1/204, CHECKPOINT_TYPE 2023-07-26 12:56:18,483 DEBUG org.apache.seatunnel.engine.server.checkpoint.CheckpointManager - Sead Operation : CheckpointBarrierTriggerOperation to [ds01]:5801 for task group:TaskGroupLocation{jobId=736433868642451457, pipelineId=1, taskGroupId=1} 2023-07-26 12:56:18,484 DEBUG org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator - Start a scheduled task to prevent checkpoint timeouts for barrier 736433868642451457/1/204, CHECKPOINT_TYPE 2023-07-26 12:56:18,484 DEBUG org.apache.seatunnel.engine.server.TaskExecutionService - [ds01]:5801 [seatunnel] [5.1] accept async execute function from TaskGroupLocation{jobId=736433868642451457, pipelineId=1, taskGroupId=1} with id 66f93122-7124-40c4-a4b8-28c1d0988bfe 2023-07-26 12:56:18,484 DEBUG org.apache.seatunnel.engine.server.checkpoint.operation.CheckpointBarrierTriggerOperation - CheckpointBarrierTriggerOperation [TaskLocation{taskGroupLocation=TaskGroupLocation{jobId=736433868642451457, pipelineId=1, taskGroupId=1}, taskID=20000, index=0}] 2023-07-26 12:56:18,484 DEBUG org.apache.seatunnel.engine.server.task.SourceSplitEnumeratorTask - split enumer trigger barrier [CheckpointBarrier 204 @ 1690347378483 Options: CHECKPOINT_TYPE] 2023-07-26 12:56:18,484 DEBUG org.apache.seatunnel.engine.server.task.SourceSplitEnumeratorTask - source split enumerator send state [HybridPendingSplitsState(snapshotPhaseState=SnapshotPhaseState(remainingTables=[], alreadyProcessedTables=[cdc_test.test_1], remainingSplits=[], assignedSplits={cdc_test.test_1:0=org.apache.seatunnel.connectors.cdc.base.source.split.SnapshotSplit@f041ba98}, splitCompletedOffsets={cdc_test.test_1:0={ts_sec=0, file=mysql-bin.000001, pos=12489317, gtids=209753da-221a-11ee-9983-76b78ab3e808:1-51389, row=0, event=0}}, isAssignerCompleted=true, isTableIdCaseSensitive=false, isRemainingTablesCheckpointed=true), incrementalPhaseState=IncrementalPhaseState())] to master 2023-07-26 12:56:18,484 DEBUG org.apache.seatunnel.engine.server.task.SourceSplitEnumeratorTask - split enumerator send to read--size: 1, location: TaskLocation{taskGroupLocation=TaskGroupLocation{jobId=736433868642451457, pipelineId=1, taskGroupId=30000}, taskID=40000, index=0}, address: [ds01]:5801 2023-07-26 12:56:18,484 DEBUG org.apache.seatunnel.engine.server.TaskExecutionService - [ds01]:5801 [seatunnel] [5.1] accept async execute function from TaskGroupLocation{jobId=736433868642451457, pipelineId=1, taskGroupId=30000} with id 89b9905a-b2b8-49b6-b5f5-69505dd7d7f4 2023-07-26 12:56:18,484 DEBUG org.apache.seatunnel.engine.server.task.operation.checkpoint.BarrierFlowOperation - CheckpointBarrierTriggerOperation [TaskLocation{taskGroupLocation=TaskGroupLocation{jobId=736433868642451457, pipelineId=1, taskGroupId=30000}, taskID=40000, index=0}] 2023-07-26 12:56:18,484 DEBUG org.apache.seatunnel.engine.server.task.flow.SourceFlowLifeCycle - source trigger barrier [CheckpointBarrier 204 @ 1690347378483 Options: CHECKPOINT_TYPE] 2023-07-26 12:56:18,484 DEBUG org.apache.seatunnel.engine.server.task.SeaTunnelTask - seatunnel task ack barrier[TaskLocation{taskGroupLocation=TaskGroupLocation{jobId=736433868642451457, pipelineId=1, taskGroupId=30000}, taskID=40000, index=0}] 2023-07-26 12:56:18,484 DEBUG org.apache.seatunnel.engine.server.task.flow.SourceFlowLifeCycle - source ack barrier finished, taskId: [40000] 2023-07-26 12:56:18,484 DEBUG org.apache.seatunnel.engine.server.task.SeaTunnelTask - seatunnel task ack barrier[TaskLocation{taskGroupLocation=TaskGroupLocation{jobId=736433868642451457, pipelineId=1, taskGroupId=30000}, taskID=40000, index=0}] 2023-07-26 12:56:18,484 DEBUG org.apache.seatunnel.engine.server.checkpoint.operation.TaskAcknowledgeOperation - TaskAcknowledgeOperation TaskLocation{taskGroupLocation=TaskGroupLocation{jobId=736433868642451457, pipelineId=1, taskGroupId=1}, taskID=20000, index=0} 2023-07-26 12:56:18,484 DEBUG org.apache.seatunnel.engine.server.checkpoint.operation.TaskAcknowledgeOperation - TaskAcknowledgeOperation TaskLocation{taskGroupLocation=TaskGroupLocation{jobId=736433868642451457, pipelineId=1, taskGroupId=30000}, taskID=40000, index=0} 2023-07-26 12:56:18,484 DEBUG org.apache.seatunnel.engine.server.checkpoint.CheckpointManager - checkpoint manager received ack TaskLocation{taskGroupLocation=TaskGroupLocation{jobId=736433868642451457, pipelineId=1, taskGroupId=1}, taskID=20000, index=0} 2023-07-26 12:56:18,484 DEBUG org.apache.seatunnel.engine.server.checkpoint.CheckpointManager - checkpoint manager received ack TaskLocation{taskGroupLocation=TaskGroupLocation{jobId=736433868642451457, pipelineId=1, taskGroupId=30000}, taskID=40000, index=0} 2023-07-26 12:56:18,484 DEBUG org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator - task[20000](1/736433868642451457) ack. CheckpointBarrier 204 @ 1690347378483 Options: CHECKPOINT_TYPE 2023-07-26 12:56:18,485 DEBUG org.apache.seatunnel.engine.server.checkpoint.PendingCheckpoint - acknowledgeTask states [[ActionSubtaskState(stateKey=ActionStateKey(name=ActionStateKey - pipeline-1 [Source[0]-MySQL-CDC-test_1_shared]), index=-1, state=[[B@4955c599])]] 2023-07-26 12:56:18,485 DEBUG org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator - task[40000](1/736433868642451457) ack. CheckpointBarrier 204 @ 1690347378483 Options: CHECKPOINT_TYPE 2023-07-26 12:56:18,485 DEBUG org.apache.seatunnel.engine.server.checkpoint.PendingCheckpoint - acknowledgeTask states [[ActionSubtaskState(stateKey=ActionStateKey(name=ActionStateKey - pipeline-1 [Source[0]-MySQL-CDC-test_1_shared]), index=0, state=[[B@21e87fe3])]] 2023-07-26 12:56:18,485 DEBUG org.apache.seatunnel.engine.server.checkpoint.operation.TaskAcknowledgeOperation - task ack finished TaskLocation{taskGroupLocation=TaskGroupLocation{jobId=736433868642451457, pipelineId=1, taskGroupId=1}, taskID=20000, index=0} 2023-07-26 12:56:18,485 DEBUG org.apache.seatunnel.engine.server.checkpoint.operation.TaskAcknowledgeOperation - task ack finished TaskLocation{taskGroupLocation=TaskGroupLocation{jobId=736433868642451457, pipelineId=1, taskGroupId=30000}, taskID=40000, index=0} 2023-07-26 12:56:18,485 DEBUG org.apache.seatunnel.engine.server.TaskExecutionService - [ds01]:5801 [seatunnel] [5.1] remove async execute function from TaskGroupLocation{jobId=736433868642451457, pipelineId=1, taskGroupId=1} with id 66f93122-7124-40c4-a4b8-28c1d0988bfe 2023-07-26 12:56:18,485 DEBUG org.apache.seatunnel.engine.server.task.flow.SourceFlowLifeCycle - send record to next finished, taskId: [40000] 2023-07-26 12:56:18,485 DEBUG org.apache.seatunnel.engine.server.TaskExecutionService - [ds01]:5801 [seatunnel] [5.1] remove async execute function from TaskGroupLocation{jobId=736433868642451457, pipelineId=1, taskGroupId=30000} with id 89b9905a-b2b8-49b6-b5f5-69505dd7d7f4 2023-07-26 12:56:18,485 DEBUG org.apache.seatunnel.engine.server.task.SeaTunnelTask - seatunnel task ack barrier[TaskLocation{taskGroupLocation=TaskGroupLocation{jobId=736433868642451457, pipelineId=1, taskGroupId=30000}, taskID=50000, index=0}] 2023-07-26 12:56:18,485 DEBUG org.apache.seatunnel.engine.server.task.SeaTunnelTask - seatunnel task ack barrier[TaskLocation{taskGroupLocation=TaskGroupLocation{jobId=736433868642451457, pipelineId=1, taskGroupId=30000}, taskID=50000, index=0}] 2023-07-26 12:56:18,485 DEBUG org.apache.seatunnel.engine.server.checkpoint.operation.TaskAcknowledgeOperation - TaskAcknowledgeOperation TaskLocation{taskGroupLocation=TaskGroupLocation{jobId=736433868642451457, pipelineId=1, taskGroupId=30000}, taskID=50000, index=0} 2023-07-26 12:56:18,485 DEBUG org.apache.seatunnel.engine.server.checkpoint.CheckpointManager - checkpoint manager received ack TaskLocation{taskGroupLocation=TaskGroupLocation{jobId=736433868642451457, pipelineId=1, taskGroupId=30000}, taskID=50000, index=0} 2023-07-26 12:56:18,485 DEBUG org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator - task[50000](1/736433868642451457) ack. CheckpointBarrier 204 @ 1690347378483 Options: CHECKPOINT_TYPE 2023-07-26 12:56:18,485 DEBUG org.apache.seatunnel.engine.server.checkpoint.PendingCheckpoint - acknowledgeTask states [[ActionSubtaskState(stateKey=ActionStateKey(name=ActionStateKey - pipeline-1 [Sink[0]-jdbc-cdc_test.test_1]), index=0, state=[])]] 2023-07-26 12:56:18,485 DEBUG org.apache.seatunnel.engine.server.checkpoint.PendingCheckpoint - checkpoint is full ack! 2023-07-26 12:56:18,485 DEBUG org.apache.seatunnel.engine.server.checkpoint.operation.TaskAcknowledgeOperation - task ack finished TaskLocation{taskGroupLocation=TaskGroupLocation{jobId=736433868642451457, pipelineId=1, taskGroupId=30000}, taskID=50000, index=0} 2023-07-26 12:56:18,485 DEBUG org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator - pending checkpoint(204/1@736433868642451457) completed! cost: 2, trigger: 1690347378483, completed: 1690347378485 2023-07-26 12:56:18,493 INFO org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator - pending checkpoint(204/1@736433868642451457) notify finished! 2023-07-26 12:56:18,493 INFO org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator - start notify checkpoint completed, checkpoint:org.apache.seatunnel.engine.server.checkpoint.CompletedCheckpoint@49dff7e 2023-07-26 12:56:18,493 DEBUG org.apache.seatunnel.engine.server.checkpoint.CheckpointManager - Sead Operation : CheckpointFinishedOperation to [ds01]:5801 for task group:TaskGroupLocation{jobId=736433868642451457, pipelineId=1, taskGroupId=1} 2023-07-26 12:56:18,493 DEBUG org.apache.seatunnel.engine.server.checkpoint.CheckpointManager - Sead Operation : CheckpointFinishedOperation to [ds01]:5801 for task group:TaskGroupLocation{jobId=736433868642451457, pipelineId=1, taskGroupId=30000} 2023-07-26 12:56:18,493 DEBUG org.apache.seatunnel.engine.server.checkpoint.CheckpointManager - Sead Operation : CheckpointFinishedOperation to [ds01]:5801 for task group:TaskGroupLocation{jobId=736433868642451457, pipelineId=1, taskGroupId=30000} 2023-07-26 12:56:21,293 DEBUG org.apache.seatunnel.engine.server.TaskExecutionService - [ds01]:5801 [seatunnel] [5.1] *********************************************** TaskExecutionServer Thread Pool Status *********************************************** activeCount : 4 threadShareTaskQueueSize : 0 completedTaskCount : 412 taskCount : 416 *********************************************** 2023-07-26 12:56:21,300 DEBUG org.apache.seatunnel.engine.server.service.slot.DefaultSlotService - start send heartbeat to resource manager, this address: [ds01]:5801 2023-07-26 12:56:21,300 DEBUG org.apache.seatunnel.engine.server.resourcemanager.AbstractResourceManager - received worker heartbeat from: [ds01]:5801 2023-07-26 12:56:26,300 DEBUG org.apache.seatunnel.engine.server.service.slot.DefaultSlotService - start send heartbeat to resource manager, this address: [ds01]:5801 2023-07-26 12:56:26,300 DEBUG org.apache.seatunnel.engine.server.resourcemanager.AbstractResourceManager - received worker heartbeat from: [ds01]:5801 2023-07-26 12:56:31,292 DEBUG org.apache.seatunnel.engine.server.TaskExecutionService - [ds01]:5801 [seatunnel] [5.1] *********************************************** TaskExecutionServer Thread Pool Status *********************************************** activeCount : 4 threadShareTaskQueueSize : 0 completedTaskCount : 412 taskCount : 416 *********************************************** 2023-07-26 12:56:31,300 DEBUG org.apache.seatunnel.engine.server.service.slot.DefaultSlotService - start send heartbeat to resource manager, this address: [ds01]:5801 2023-07-26 12:56:31,300 DEBUG org.apache.seatunnel.engine.server.resourcemanager.AbstractResourceManager - received worker heartbeat from: [ds01]:5801 2023-07-26 12:56:33,483 INFO org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator - wait checkpoint completed: 205 ``` ### SeaTunnel Version 2.3.2 ### SeaTunnel Config ```conf seatunnel: engine: backup-count: 1 queue-type: blockingqueue print-execution-info-interval: 60 print-job-metrics-info-interval: 60 slot-service: dynamic-slot: true checkpoint: interval: 10000 timeout: 60000 max-concurrent: 1 tolerable-failure: 2 storage: type: hdfs max-retained: 3 plugin-config: namespace: /tmp/seatunnel/checkpoint_snapshot storage.type: hdfs fs.defaultFS: file:///tmp ``` ### Running Command ```shell ./bin/seatunnel-cluster.sh -d ``` ### Error Exception ```log Nothing ``` ### Flink or Spark Version _No response_ ### Java or Scala Version 1.8.0_191 ### Screenshots _No response_ ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
