potterhe created FLINK-38061:
--------------------------------
Summary: Caused by: java.util.ConcurrentModificationException When
the MySql Chunk Split stage triggers a checkpoint
Key: FLINK-38061
URL: https://issues.apache.org/jira/browse/FLINK-38061
Project: Flink
Issue Type: Bug
Components: Flink CDC
Affects Versions: cdc-3.4.0
Reporter: potterhe
{code:java}
2025-06-19 15:17:22,914 INFO
org.apache.flink.cdc.connectors.mysql.source.assigners.MySqlChunkSplitter [] -
The actual distribution factor for table
mall_oms_sharding_3.oms_performance_serve_detail_sharding_127 is
2.3258008172726633E13, the lower bound of evenly distribution factor is 0.05,
the upper bound of evenly distribution factor is 1000.0
2025-06-19 15:17:22,914 INFO
org.apache.flink.cdc.connectors.mysql.source.assigners.MySqlChunkSplitter [] -
Use unevenly-sized chunks for table
mall_oms_sharding_3.oms_performance_serve_detail_sharding_127, the chunk size
is 50000 from null
2025-06-19 15:17:23,002 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Triggering
checkpoint 1 (type=CheckpointType{name='Checkpoint',
sharingFilesStrategy=FORWARD_BACKWARD}) @ 1750317442997 for job
f44cd9bc7a4c8eeec0aa28cb959dcfcf.
2025-06-19 15:17:23,034 INFO
org.apache.flink.cdc.connectors.mysql.source.assigners.MySqlChunkSplitter [] -
ChunkSplitter has split 0 chunks for table
mall_oms_sharding_3.oms_performance_serve_detail_sharding_127
2025-06-19 15:17:23,034 INFO
org.apache.flink.cdc.connectors.mysql.source.assigners.MySqlSnapshotSplitAssigner
[] - Split table mall_oms_sharding_3.oms_performance_serve_detail_sharding_127
into 1 chunks, time cost: 146ms.
2025-06-19 15:17:23,034 INFO
org.apache.flink.cdc.connectors.mysql.source.assigners.MySqlSnapshotSplitAssigner
[] - Start splitting table
mall_oms_sharding_3.oms_performance_serve_detail_sharding_96 into chunks...
2025-06-19 15:17:23,059 INFO
org.apache.flink.cdc.connectors.mysql.source.assigners.MySqlChunkSplitter [] -
The distribution factor of table
mall_oms_sharding_3.oms_performance_serve_detail_sharding_96 is
2.358148187707668E13 according to the min split key 612073909416749057, max
split key 812115620179990530 and approximate row count 8483
2025-06-19 15:17:23,059 INFO
org.apache.flink.cdc.connectors.mysql.source.assigners.MySqlChunkSplitter [] -
The actual distribution factor for table
mall_oms_sharding_3.oms_performance_serve_detail_sharding_96 is
2.358148187707668E13, the lower bound of evenly distribution factor is 0.05,
the upper bound of evenly distribution factor is 1000.0
2025-06-19 15:17:23,059 INFO
org.apache.flink.cdc.connectors.mysql.source.assigners.MySqlChunkSplitter [] -
Use unevenly-sized chunks for table
mall_oms_sharding_3.oms_performance_serve_detail_sharding_96, the chunk size is
50000 from null
2025-06-19 15:17:23,135 INFO
org.apache.flink.cdc.runtime.operators.schema.common.SchemaRegistry [] - Going
to start checkpoint No.1
2025-06-19 15:17:23,181 INFO
org.apache.flink.cdc.connectors.mysql.source.assigners.MySqlChunkSplitter [] -
ChunkSplitter has split 0 chunks for table
mall_oms_sharding_3.oms_performance_serve_detail_sharding_96
2025-06-19 15:17:23,181 INFO
org.apache.flink.cdc.connectors.mysql.source.assigners.MySqlSnapshotSplitAssigner
[] - Split table mall_oms_sharding_3.oms_performance_serve_detail_sharding_96
into 1 chunks, time cost: 147ms.
2025-06-19 15:17:23,181 INFO
org.apache.flink.cdc.connectors.mysql.source.assigners.MySqlSnapshotSplitAssigner
[] - Start splitting table
mall_oms_sharding_3.oms_performance_serve_detail_sharding_97 into chunks...
2025-06-19 15:17:23,208 INFO
org.apache.flink.cdc.connectors.mysql.source.assigners.MySqlChunkSplitter [] -
The distribution factor of table
mall_oms_sharding_3.oms_performance_serve_detail_sharding_97 is
1.9248231912303316E13 according to the min split key 612073859831687169, max
split key 812332464647290882 and approximate row count 10404
2025-06-19 15:17:23,208 INFO
org.apache.flink.cdc.connectors.mysql.source.assigners.MySqlChunkSplitter [] -
The actual distribution factor for table
mall_oms_sharding_3.oms_performance_serve_detail_sharding_97 is
1.9248231912303316E13, the lower bound of evenly distribution factor is 0.05,
the upper bound of evenly distribution factor is 1000.0
2025-06-19 15:17:23,208 INFO
org.apache.flink.cdc.connectors.mysql.source.assigners.MySqlChunkSplitter [] -
Use unevenly-sized chunks for table
mall_oms_sharding_3.oms_performance_serve_detail_sharding_97, the chunk size is
50000 from null
2025-06-19 15:17:23,323 INFO
org.apache.flink.cdc.connectors.mysql.source.assigners.MySqlChunkSplitter [] -
ChunkSplitter has split 0 chunks for table
mall_oms_sharding_3.oms_performance_serve_detail_sharding_97
2025-06-19 15:17:23,323 INFO
org.apache.flink.cdc.connectors.mysql.source.assigners.MySqlSnapshotSplitAssigner
[] - Split table mall_oms_sharding_3.oms_performance_serve_detail_sharding_97
into 1 chunks, time cost: 142ms.
2025-06-19 15:17:23,323 INFO
org.apache.flink.cdc.connectors.mysql.source.assigners.MySqlSnapshotSplitAssigner
[] - Start splitting table
mall_oms_sharding_3.oms_performance_serve_detail_sharding_98 into chunks...
2025-06-19 15:17:23,351 INFO
org.apache.flink.cdc.connectors.mysql.source.assigners.MySqlChunkSplitter [] -
The distribution factor of table
mall_oms_sharding_3.oms_performance_serve_detail_sharding_98 is
2.72202276885709E13 according to the min split key 612074076756895745, max
split key 812115530040203266 and approximate row count 7349
2025-06-19 15:17:23,351 INFO
org.apache.flink.cdc.connectors.mysql.source.assigners.MySqlChunkSplitter [] -
The actual distribution factor for table
mall_oms_sharding_3.oms_performance_serve_detail_sharding_98 is
2.72202276885709E13, the lower bound of evenly distribution factor is 0.05, the
upper bound of evenly distribution factor is 1000.0
2025-06-19 15:17:23,351 INFO
org.apache.flink.cdc.connectors.mysql.source.assigners.MySqlChunkSplitter [] -
Use unevenly-sized chunks for table
mall_oms_sharding_3.oms_performance_serve_detail_sharding_98, the chunk size is
50000 from null
2025-06-19 15:17:23,476 INFO
org.apache.flink.cdc.connectors.mysql.source.assigners.MySqlChunkSplitter [] -
ChunkSplitter has split 0 chunks for table
mall_oms_sharding_3.oms_performance_serve_detail_sharding_98
2025-06-19 15:17:23,477 INFO
org.apache.flink.cdc.connectors.mysql.source.assigners.MySqlSnapshotSplitAssigner
[] - Split table mall_oms_sharding_3.oms_performance_serve_detail_sharding_98
into 1 chunks, time cost: 154ms.
2025-06-19 15:17:23,477 INFO
org.apache.flink.cdc.connectors.mysql.source.assigners.MySqlSnapshotSplitAssigner
[] - Start splitting table
mall_oms_sharding_3.oms_performance_serve_detail_sharding_99 into chunks...
2025-06-19 15:17:23,479 WARN
org.apache.flink.runtime.checkpoint.CheckpointFailureManager [] - Failed to
trigger or complete checkpoint 1 for job f44cd9bc7a4c8eeec0aa28cb959dcfcf. (0
consecutive failed attempts so far)
org.apache.flink.runtime.checkpoint.CheckpointException: Trigger checkpoint
failure.
at
org.apache.flink.runtime.checkpoint.CheckpointCoordinator.lambda$getCheckpointException$17(CheckpointCoordinator.java:2365)
~[flink-dist-1.20.1.jar:1.20.1]
at java.util.Optional.orElseGet(Unknown Source) ~[?:?]
at
org.apache.flink.runtime.checkpoint.CheckpointCoordinator.getCheckpointException(CheckpointCoordinator.java:2364)
~[flink-dist-1.20.1.jar:1.20.1]
at
org.apache.flink.runtime.checkpoint.CheckpointCoordinator.onTriggerFailure(CheckpointCoordinator.java:1056)
~[flink-dist-1.20.1.jar:1.20.1]
at
org.apache.flink.runtime.checkpoint.CheckpointCoordinator.onTriggerFailure(CheckpointCoordinator.java:1034)
~[flink-dist-1.20.1.jar:1.20.1]
at
org.apache.flink.runtime.checkpoint.CheckpointCoordinator.lambda$startTriggeringCheckpoint$7(CheckpointCoordinator.java:760)
~[flink-dist-1.20.1.jar:1.20.1]
at java.util.concurrent.CompletableFuture.uniHandle(Unknown Source) ~[?:?]
at java.util.concurrent.CompletableFuture$UniHandle.tryFire(Unknown Source)
~[?:?]
at java.util.concurrent.CompletableFuture$Completion.run(Unknown Source) ~[?:?]
at org.apache.flink.util.MdcUtils.lambda$wrapRunnable$1(MdcUtils.java:70)
~[flink-dist-1.20.1.jar:1.20.1]
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) [?:?]
at java.util.concurrent.FutureTask.run(Unknown Source) [?:?]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown
Source) [?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:?]
at java.lang.Thread.run(Unknown Source) [?:?]
Caused by: java.util.ConcurrentModificationException
at java.util.HashMap$HashIterator.nextNode(Unknown Source) ~[?:?]
at java.util.HashMap$EntryIterator.next(Unknown Source) ~[?:?]
at java.util.HashMap$EntryIterator.next(Unknown Source) ~[?:?]
at
org.apache.flink.cdc.connectors.mysql.source.split.MySqlSplitSerializer.writeTableSchemas(MySqlSplitSerializer.java:194)
~[flink-cdc-pipeline-connector-mysql-3.4.0.jar:3.5-SNAPSHOT]
at
org.apache.flink.cdc.connectors.mysql.source.assigners.state.PendingSplitsStateSerializer.serializeSnapshotPendingSplitsState(PendingSplitsStateSerializer.java:165)
~[flink-cdc-pipeline-connector-mysql-3.4.0.jar:3.5-SNAPSHOT]
at
org.apache.flink.cdc.connectors.mysql.source.assigners.state.PendingSplitsStateSerializer.serializeHybridPendingSplitsState(PendingSplitsStateSerializer.java:182)
~[flink-cdc-pipeline-connector-mysql-3.4.0.jar:3.5-SNAPSHOT]
at
org.apache.flink.cdc.connectors.mysql.source.assigners.state.PendingSplitsStateSerializer.serialize(PendingSplitsStateSerializer.java:85)
~[flink-cdc-pipeline-connector-mysql-3.4.0.jar:3.5-SNAPSHOT]
at
org.apache.flink.cdc.connectors.mysql.source.assigners.state.PendingSplitsStateSerializer.serialize(PendingSplitsStateSerializer.java:45)
~[flink-cdc-pipeline-connector-mysql-3.4.0.jar:3.5-SNAPSHOT]
at
org.apache.flink.runtime.source.coordinator.SourceCoordinator.writeCheckpointBytes(SourceCoordinator.java:594)
~[flink-dist-1.20.1.jar:1.20.1]
at
org.apache.flink.runtime.source.coordinator.SourceCoordinator.toBytes(SourceCoordinator.java:579)
~[flink-dist-1.20.1.jar:1.20.1]
at
org.apache.flink.runtime.source.coordinator.SourceCoordinator.lambda$checkpointCoordinator$7(SourceCoordinator.java:424)
~[flink-dist-1.20.1.jar:1.20.1]
at
org.apache.flink.runtime.source.coordinator.SourceCoordinator.lambda$runInEventLoop$10(SourceCoordinator.java:533)
~[flink-dist-1.20.1.jar:1.20.1]
at
org.apache.flink.util.ThrowableCatchingRunnable.run(ThrowableCatchingRunnable.java:40)
~[flink-dist-1.20.1.jar:1.20.1]
... 6 more
2025-06-19 15:17:23,484 INFO
org.apache.flink.runtime.source.coordinator.SourceCoordinator [] - Marking
checkpoint 1 as aborted for source Source: Flink CDC Event Source: mysql. {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
