Python-IBM opened a new issue, #5688: URL: https://github.com/apache/seatunnel/issues/5688
### Search before asking - [X] I had searched in the [issues](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues. ### What happened In cluster mode, there are three machines that can access port 5801, but sometimes exceptions may occur and one of them cannot be heard. ### SeaTunnel Version seatunnel-2.3.3 ### SeaTunnel Config ```conf 1、hazelcast-client.yaml hazelcast-client: cluster-name: seatunnel properties: hazelcast.logging.type: log4j2 network: cluster-members: - 10.10.0.22:5801 - 10.10.0.27:5801 - 10.10.0.30:5801 2、hazelcast.yaml hazelcast: cluster-name: seatunnel network: rest-api: enabled: true endpoint-groups: CLUSTER_WRITE: enabled: true DATA: enabled: true join: tcp-ip: enabled: true member-list: - 10.10.0.22 - 10.10.0.27 - 10.10.0.30 port: auto-increment: false port: 5801 properties: hazelcast.invocation.max.retry.count: 20 hazelcast.tcp.join.port.try.count: 30 hazelcast.logging.type: log4j2 hazelcast.operation.generic.thread.count: 50 3、seatunnel.yaml seatunnel: engine: history-job-expire-minutes: 1440 backup-count: 2 queue-type: blockingqueue print-execution-info-interval: 60 print-job-metrics-info-interval: 60 slot-service: dynamic-slot: true checkpoint: interval: 10000 timeout: 60000 storage: type: hdfs max-retained: 3 plugin-config: namespace: /tmp/seatunnel/checkpoint_snapshot storage.type: hdfs fs.defaultFS: file:///tmp/ ``` ### Running Command ```shell 1、Start Cluster Service sh seatunnel-cluster.sh 2、Perform configuration tasks ${SEATUNNEL_HOME}/bin/seatunnel.sh --config seatunnel_4_15.conf --deploy-mode cluster ``` ### Error Exception ```log [INFO] 2023-10-23 16:38:38.182 +0800 - -> 2023-10-23 16:38:37,399 INFO org.apache.seatunnel.engine.client.job.JobMetricsRunner - *********************************************** Job Progress Information *********************************************** Job Id : 768752386423390211 Read Count So Far : 4802837 Write Count So Far : 4801954 Average Read Count : 7546/s Average Write Count : 7531/s Last Statistic Time : 2023-10-23 16:37:36 Current Statistic Time : 2023-10-23 16:38:37 *********************************************** [INFO] 2023-10-23 16:39:55.451 +0800 - -> 2023-10-23 16:39:51,586 WARN com.hazelcast.client.impl.connection.tcp.HeartbeatManager - hz.client_1 [seatunnel] [5.1] Heartbeat failed over the connection: ClientConnection{alive=true, connectionId=2, channel=NioChannel{/10.10.0.22:44703->/10.10.0.22:5801}, remoteAddress=[10.10.0.22]:5801, lastReadTime=2023-10-23 16:38:46.649, lastWriteTime=2023-10-23 16:39:45.832, closedTime=never, connected server version=5.1} 2023-10-23 16:39:51,588 WARN com.hazelcast.client.impl.connection.tcp.TcpClientConnection - hz.client_1 [seatunnel] [5.1] ClientConnection{alive=false, connectionId=2, channel=NioChannel{/10.10.0.22:44703->/10.10.0.22:5801}, remoteAddress=[10.10.0.22]:5801, lastReadTime=2023-10-23 16:38:46.649, lastWriteTime=2023-10-23 16:39:45.832, closedTime=2023-10-23 16:39:51.587, connected server version=5.1} closed. Reason: Heartbeat timed out com.hazelcast.spi.exception.TargetDisconnectedException: Heartbeat timed out to connection ClientConnection{alive=true, connectionId=2, channel=NioChannel{/10.10.0.22:44703->/10.10.0.22:5801}, remoteAddress=[10.10.0.22]:5801, lastReadTime=2023-10-23 16:38:46.649, lastWriteTime=2023-10-23 16:39:45.832, closedTime=never, connected server version=5.1} at com.hazelcast.client.impl.connection.tcp.HeartbeatManager.lambda$start$0(HeartbeatManager.java:59) ~[seatunnel-starter.jar:2.3.3] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_381] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_381] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_381] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_381] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_381] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_381] at java.lang.Thread.run(Thread.java:750) [?:1.8.0_381] at com.hazelcast.internal.util.executor.HazelcastManagedThread.executeRun(HazelcastManagedThread.java:76) [seatunnel-starter.jar:2.3.3] at com.hazelcast.internal.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:102) [seatunnel-starter.jar:2.3.3] 2023-10-23 16:39:52,541 INFO com.hazelcast.client.impl.spi.ClientClusterService - hz.client_1 [seatunnel] [5.1] Members [2] { Member [10.10.0.27]:5801 - 410ba9af-8faa-4d99-a014-cc7b8f507026 Member [10.10.0.30]:5801 - f1bdfdd3-1135-45d8-97a2-31a2431ea6b7 } [INFO] 2023-10-23 16:40:00.149 +0800 - -> 2023-10-23 16:39:59,667 INFO org.apache.seatunnel.engine.client.job.ClientJobProxy - Job (768752386423390211) end with unknown state, and throw Exception: java.util.concurrent.ExecutionException: com.hazelcast.core.OperationTimeoutException: ClientInvocation{clientMessage = ClientMessage{connection=null, length=30, operation=SeaTunnel.WaitForJobComplete, isRetryable=true, correlationId=16, messageType=de0300, isEvent=false, isfragmented=false}, objectName = null, target = uuid 15e6f0b8-d596-4824-a903-12ce660ca55d, sentConnection = null} timed out because exception occurred after client invocation timeout 120000 ms. Current time: 2023-10-23 16:39:54.370. Start time: 2023-10-23 16:27:36.878. Total elapsed time: 737492 ms. at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908) at org.apache.seatunnel.engine.client.job.ClientJobProxy.lambda$waitForJobComplete$0(ClientJobProxy.java:101) at org.apache.seatunnel.common.utils.RetryUtils.retryWithException(RetryUtils.java:48) at org.apache.seatunnel.engine.client.job.ClientJobProxy.waitForJobComplete(ClientJobProxy.java:97) at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:184) at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40) at org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:34) Caused by: com.hazelcast.core.OperationTimeoutException: ClientInvocation{clientMessage = ClientMessage{connection=null, length=30, operation=SeaTunnel.WaitForJobComplete, isRetryable=true, correlationId=16, messageType=de0300, isEvent=false, isfragmented=false}, objectName = null, target = uuid 15e6f0b8-d596-4824-a903-12ce660ca55d, sentConnection = null} timed out because exception occurred after client invocation timeout 120000 ms. Current time: 2023-10-23 16:39:54.370. Start time: 2023-10-23 16:27:36.878. Total elapsed time: 737492 ms. at com.hazelcast.client.impl.spi.impl.ClientInvocation.notifyExceptionWithOwnedPermission(ClientInvocation.java:339) at com.hazelcast.client.impl.spi.impl.ClientInvocationServiceImpl.onConnectionClose(ClientInvocationServiceImpl.java:221) at com.hazelcast.client.impl.connection.tcp.TcpClientConnectionManager.onConnectionClose(TcpClientConnectionManager.java:758) at com.hazelcast.client.impl.connection.tcp.TcpClientConnection.close(TcpClientConnection.java:191) at com.hazelcast.client.impl.connection.tcp.HeartbeatManager.lambda$start$0(HeartbeatManager.java:59) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) at com.hazelcast.internal.util.executor.HazelcastManagedThread.executeRun(HazelcastManagedThread.java:76) at com.hazelcast.internal.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:102) Caused by: com.hazelcast.spi.exception.TargetDisconnectedException: Heartbeat timed out at com.hazelcast.client.impl.spi.impl.ClientInvocationServiceImpl.onConnectionClose(ClientInvocationServiceImpl.java:220) ... 12 more Caused by: com.hazelcast.spi.exception.TargetDisconnectedException: Heartbeat timed out to connection ClientConnection{alive=true, connectionId=2, channel=NioChannel{/10.10.0.22:44703->/10.10.0.22:5801}, remoteAddress=[10.10.0.22]:5801, lastReadTime=2023-10-23 16:38:46.649, lastWriteTime=2023-10-23 16:39:45.832, closedTime=never, connected server version=5.1} ... 10 more 2023-10-23 16:39:59,668 INFO com.hazelcast.core.LifecycleService - hz.client_1 [seatunnel] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is SHUTTING_DOWN [INFO] 2023-10-23 16:40:01.467 +0800 - -> 2023-10-23 16:40:01,339 INFO com.hazelcast.client.impl.connection.ClientConnectionManager - hz.client_1 [seatunnel] [5.1] Removed connection to endpoint: [10.10.0.22]:5801:15e6f0b8-d596-4824-a903-12ce660ca55d, connection: ClientConnection{alive=false, connectionId=2, channel=NioChannel{/10.10.0.22:44703->/10.10.0.22:5801}, remoteAddress=[10.10.0.22]:5801, lastReadTime=2023-10-23 16:38:46.649, lastWriteTime=2023-10-23 16:39:45.832, closedTime=2023-10-23 16:39:51.587, connected server version=5.1} [INFO] 2023-10-23 16:40:21.989 +0800 - -> 2023-10-23 16:40:21,874 INFO com.hazelcast.client.impl.connection.ClientConnectionManager - hz.client_1 [seatunnel] [5.1] Removed connection to endpoint: [10.10.0.30]:5801:f1bdfdd3-1135-45d8-97a2-31a2431ea6b7, connection: ClientConnection{alive=false, connectionId=3, channel=NioChannel{/10.10.0.22:47851->/10.10.0.30:5801}, remoteAddress=[10.10.0.30]:5801, lastReadTime=2023-10-23 16:39:58.637, lastWriteTime=2023-10-23 16:39:59.209, closedTime=2023-10-23 16:40:01.338, connected server version=5.1} [INFO] 2023-10-23 16:40:23.320 +0800 - -> 2023-10-23 16:40:22,755 INFO com.hazelcast.client.impl.connection.ClientConnectionManager - hz.client_1 [seatunnel] [5.1] Removed connection to endpoint: [10.10.0.27]:5801:410ba9af-8faa-4d99-a014-cc7b8f507026, connection: ClientConnection{alive=false, connectionId=1, channel=NioChannel{/10.10.0.22:54089->/10.10.0.27:5801}, remoteAddress=[10.10.0.27]:5801, lastReadTime=2023-10-23 16:40:21.770, lastWriteTime=2023-10-23 16:40:21.869, closedTime=2023-10-23 16:40:21.874, connected server version=5.1} 2023-10-23 16:40:22,755 INFO com.hazelcast.core.LifecycleService - hz.client_1 [seatunnel] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is CLIENT_DISCONNECTED 2023-10-23 16:40:22,777 INFO com.hazelcast.core.LifecycleService - hz.client_1 [seatunnel] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is SHUTDOWN [INFO] 2023-10-23 16:40:28.397 +0800 - -> 2023-10-23 16:40:28,347 WARN org.apache.seatunnel.engine.client.job.JobMetricsRunner - Failed to get job metrics summary, it maybe first-run [INFO] 2023-10-23 16:40:29.699 +0800 - -> 2023-10-23 16:40:28,733 INFO org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand - Closed SeaTunnel client...... 2023-10-23 16:40:28,734 INFO org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand - Closed metrics executor service ...... 2023-10-23 16:40:28,734 ERROR org.apache.seatunnel.core.starter.SeaTunnel - =============================================================================== 2023-10-23 16:40:28,734 ERROR org.apache.seatunnel.core.starter.SeaTunnel - Fatal Error, 2023-10-23 16:40:28,734 ERROR org.apache.seatunnel.core.starter.SeaTunnel - Please submit bug report in https://github.com/apache/seatunnel/issues 2023-10-23 16:40:28,734 ERROR org.apache.seatunnel.core.starter.SeaTunnel - Reason:SeaTunnel job executed failed 2023-10-23 16:40:28,738 ERROR org.apache.seatunnel.core.starter.SeaTunnel - Exception StackTrace:org.apache.seatunnel.core.starter.exception.CommandExecuteException: SeaTunnel job executed failed at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:191) at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40) at org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:34) Caused by: java.lang.RuntimeException: java.util.concurrent.ExecutionException: com.hazelcast.core.OperationTimeoutException: ClientInvocation{clientMessage = ClientMessage{connection=null, length=30, operation=SeaTunnel.WaitForJobComplete, isRetryable=true, correlationId=16, messageType=de0300, isEvent=false, isfragmented=false}, objectName = null, target = uuid 15e6f0b8-d596-4824-a903-12ce660ca55d, sentConnection = null} timed out because exception occurred after client invocation timeout 120000 ms. Current time: 2023-10-23 16:39:54.370. Start time: 2023-10-23 16:27:36.878. Total elapsed time: 737492 ms. at org.apache.seatunnel.engine.client.job.ClientJobProxy.waitForJobComplete(ClientJobProxy.java:117) at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:184) ... 2 more Caused by: java.util.concurrent.ExecutionException: com.hazelcast.core.OperationTimeoutException: ClientInvocation{clientMessage = ClientMessage{connection=null, length=30, operation=SeaTunnel.WaitForJobComplete, isRetryable=true, correlationId=16, messageType=de0300, isEvent=false, isfragmented=false}, objectName = null, target = uuid 15e6f0b8-d596-4824-a903-12ce660ca55d, sentConnection = null} timed out because exception occurred after client invocation timeout 120000 ms. Current time: 2023-10-23 16:39:54.370. Start time: 2023-10-23 16:27:36.878. Total elapsed time: 737492 ms. at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908) at org.apache.seatunnel.engine.client.job.ClientJobProxy.lambda$waitForJobComplete$0(ClientJobProxy.java:101) at org.apache.seatunnel.common.utils.RetryUtils.retryWithException(RetryUtils.java:48) at org.apache.seatunnel.engine.client.job.ClientJobProxy.waitForJobComplete(ClientJobProxy.java:97) ... 3 more Caused by: com.hazelcast.core.OperationTimeoutException: ClientInvocation{clientMessage = ClientMessage{connection=null, length=30, operation=SeaTunnel.WaitForJobComplete, isRetryable=true, correlationId=16, messageType=de0300, isEvent=false, isfragmented=false}, objectName = null, target = uuid 15e6f0b8-d596-4824-a903-12ce660ca55d, sentConnection = null} timed out because exception occurred after client invocation timeout 120000 ms. Current time: 2023-10-23 16:39:54.370. Start time: 2023-10-23 16:27:36.878. Total elapsed time: 737492 ms. at com.hazelcast.client.impl.spi.impl.ClientInvocation.notifyExceptionWithOwnedPermission(ClientInvocation.java:339) at com.hazelcast.client.impl.spi.impl.ClientInvocationServiceImpl.onConnectionClose(ClientInvocationServiceImpl.java:221) at com.hazelcast.client.impl.connection.tcp.TcpClientConnectionManager.onConnectionClose(TcpClientConnectionManager.java:758) at com.hazelcast.client.impl.connection.tcp.TcpClientConnection.close(TcpClientConnection.java:191) at com.hazelcast.client.impl.connection.tcp.HeartbeatManager.lambda$start$0(HeartbeatManager.java:59) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) at com.hazelcast.internal.util.executor.HazelcastManagedThread.executeRun(HazelcastManagedThread.java:76) at com.hazelcast.internal.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:102) Caused by: com.hazelcast.spi.exception.TargetDisconnectedException: Heartbeat timed out at com.hazelcast.client.impl.spi.impl.ClientInvocationServiceImpl.onConnectionClose(ClientInvocationServiceImpl.java:220) ... 12 more Caused by: com.hazelcast.spi.exception.TargetDisconnectedException: Heartbeat timed out to connection ClientConnection{alive=true, connectionId=2, channel=NioChannel{/10.10.0.22:44703->/10.10.0.22:5801}, remoteAddress=[10.10.0.22]:5801, lastReadTime=2023-10-23 16:38:46.649, lastWriteTime=2023-10-23 16:39:45.832, closedTime=never, connected server version=5.1} ... 10 more 2023-10-23 16:40:28,738 ERROR org.apache.seatunnel.core.starter.SeaTunnel - =============================================================================== Exception in thread "main" org.apache.seatunnel.core.starter.exception.CommandExecuteException: SeaTunnel job executed failed at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:191) at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40) at org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:34) Caused by: java.lang.RuntimeException: java.util.concurrent.ExecutionException: com.hazelcast.core.OperationTimeoutException: ClientInvocation{clientMessage = ClientMessage{connection=null, length=30, operation=SeaTunnel.WaitForJobComplete, isRetryable=true, correlationId=16, messageType=de0300, isEvent=false, isfragmented=false}, objectName = null, target = uuid 15e6f0b8-d596-4824-a903-12ce660ca55d, sentConnection = null} timed out because exception occurred after client invocation timeout 120000 ms. Current time: 2023-10-23 16:39:54.370. Start time: 2023-10-23 16:27:36.878. Total elapsed time: 737492 ms. at org.apache.seatunnel.engine.client.job.ClientJobProxy.waitForJobComplete(ClientJobProxy.java:117) at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:184) ... 2 more Caused by: java.util.concurrent.ExecutionException: com.hazelcast.core.OperationTimeoutException: ClientInvocation{clientMessage = ClientMessage{connection=null, length=30, operation=SeaTunnel.WaitForJobComplete, isRetryable=true, correlationId=16, messageType=de0300, isEvent=false, isfragmented=false}, objectName = null, target = uuid 15e6f0b8-d596-4824-a903-12ce660ca55d, sentConnection = null} timed out because exception occurred after client invocation timeout 120000 ms. Current time: 2023-10-23 16:39:54.370. Start time: 2023-10-23 16:27:36.878. Total elapsed time: 737492 ms. at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908) at org.apache.seatunnel.engine.client.job.ClientJobProxy.lambda$waitForJobComplete$0(ClientJobProxy.java:101) at org.apache.seatunnel.common.utils.RetryUtils.retryWithException(RetryUtils.java:48) at org.apache.seatunnel.engine.client.job.ClientJobProxy.waitForJobComplete(ClientJobProxy.java:97) ... 3 more Caused by: com.hazelcast.core.OperationTimeoutException: ClientInvocation{clientMessage = ClientMessage{connection=null, length=30, operation=SeaTunnel.WaitForJobComplete, isRetryable=true, correlationId=16, messageType=de0300, isEvent=false, isfragmented=false}, objectName = null, target = uuid 15e6f0b8-d596-4824-a903-12ce660ca55d, sentConnection = null} timed out because exception occurred after client invocation timeout 120000 ms. Current time: 2023-10-23 16:39:54.370. Start time: 2023-10-23 16:27:36.878. Total elapsed time: 737492 ms. at com.hazelcast.client.impl.spi.impl.ClientInvocation.notifyExceptionWithOwnedPermission(ClientInvocation.java:339) at com.hazelcast.client.impl.spi.impl.ClientInvocationServiceImpl.onConnectionClose(ClientInvocationServiceImpl.java:221) at com.hazelcast.client.impl.connection.tcp.TcpClientConnectionManager.onConnectionClose(TcpClientConnectionManager.java:758) at com.hazelcast.client.impl.connection.tcp.TcpClientConnection.close(TcpClientConnection.java:191) at com.hazelcast.client.impl.connection.tcp.HeartbeatManager.lambda$start$0(HeartbeatManager.java:59) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) at com.hazelcast.internal.util.executor.HazelcastManagedThread.executeRun(HazelcastManagedThread.java:76) at com.hazelcast.internal.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:102) Caused by: com.hazelcast.spi.exception.TargetDisconnectedException: Heartbeat timed out at com.hazelcast.client.impl.spi.impl.ClientInvocationServiceImpl.onConnectionClose(ClientInvocationServiceImpl.java:220) ... 12 more Caused by: com.hazelcast.spi.exception.TargetDisconnectedException: Heartbeat timed out to connection ClientConnection{alive=true, connectionId=2, channel=NioChannel{/10.10.0.22:44703->/10.10.0.22:5801}, remoteAddress=[10.10.0.22]:5801, lastReadTime=2023-10-23 16:38:46.649, lastWriteTime=2023-10-23 16:39:45.832, closedTime=never, connected server version=5.1} ... 10 more [INFO] 2023-10-23 16:40:33.523 +0800 - -> 2023-10-23 16:40:32,678 INFO org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand - run shutdown hook because get close signal 2023-10-23 16:40:32,679 WARN org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand - Task will be closed due to client shutdown. [INFO] 2023-10-23 16:40:35.465 +0800 - -> 2023-10-23 16:40:33,805 ERROR org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand - Cancel job failed. java.util.concurrent.ExecutionException: com.hazelcast.client.HazelcastClientNotActiveException: Client is shutting down at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) ~[?:1.8.0_381] at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1928) ~[?:1.8.0_381] at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.lambda$execute$1(ClientExecuteCommand.java:164) ~[seatunnel-starter.jar:2.3.3] at java.lang.Thread.run(Thread.java:750) [?:1.8.0_381] Caused by: com.hazelcast.client.HazelcastClientNotActiveException: Client is shutting down at com.hazelcast.client.impl.spi.impl.ClientInvocation.notifyExceptionWithOwnedPermission(ClientInvocation.java:316) ~[seatunnel-starter.jar:2.3.3] at com.hazelcast.client.impl.spi.impl.ClientInvocation.invokeOnSelection(ClientInvocation.java:209) ~[seatunnel-starter.jar:2.3.3] at com.hazelcast.client.impl.spi.impl.ClientInvocation.invoke(ClientInvocation.java:156) ~[seatunnel-starter.jar:2.3.3] at org.apache.seatunnel.engine.client.SeaTunnelHazelcastClient.requestAndGetCompletableFuture(SeaTunnelHazelcastClient.java:123) ~[seatunnel-starter.jar:2.3.3] at org.apache.seatunnel.engine.client.SeaTunnelHazelcastClient.requestOnMasterAndGetCompletableFuture(SeaTunnelHazelcastClient.java:132) ~[seatunnel-starter.jar:2.3.3] at org.apache.seatunnel.engine.client.job.ClientJobProxy.cancelJob(ClientJobProxy.java:148) ~[seatunnel-starter.jar:2.3.3] at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.shutdownHook(ClientExecuteCommand.java:254) ~[seatunnel-starter.jar:2.3.3] at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.lambda$null$0(ClientExecuteCommand.java:161) ~[seatunnel-starter.jar:2.3.3] at java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1640) ~[?:1.8.0_381] at java.util.concurrent.CompletableFuture$AsyncRun.exec(CompletableFuture.java:1632) ~[?:1.8.0_381] at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) ~[?:1.8.0_381] at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1067) ~[?:1.8.0_381] at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1703) ~[?:1.8.0_381] at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:172) ~[?:1.8.0_381] Caused by: java.io.IOException: No connection found to cluster. at com.hazelcast.client.impl.connection.tcp.TcpClientConnectionManager.checkInvocationAllowed(TcpClientConnectionManager.java:548) ~[seatunnel-starter.jar:2.3.3] at com.hazelcast.client.impl.spi.impl.ClientInvocationServiceImpl.checkInvocationAllowed(ClientInvocationServiceImpl.java:294) ~[seatunnel-starter.jar:2.3.3] at com.hazelcast.client.impl.spi.impl.ClientInvocation.invokeOnSelection(ClientInvocation.java:178) ~[seatunnel-starter.jar:2.3.3] at com.hazelcast.client.impl.spi.impl.ClientInvocation.invoke(ClientInvocation.java:156) ~[seatunnel-starter.jar:2.3.3] at org.apache.seatunnel.engine.client.SeaTunnelHazelcastClient.requestAndGetCompletableFuture(SeaTunnelHazelcastClient.java:123) ~[seatunnel-starter.jar:2.3.3] at org.apache.seatunnel.engine.client.SeaTunnelHazelcastClient.requestOnMasterAndGetCompletableFuture(SeaTunnelHazelcastClient.java:132) ~[seatunnel-starter.jar:2.3.3] at org.apache.seatunnel.engine.client.job.ClientJobProxy.cancelJob(ClientJobProxy.java:148) ~[seatunnel-starter.jar:2.3.3] at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.shutdownHook(ClientExecuteCommand.java:254) ~[seatunnel-starter.jar:2.3.3] at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.lambda$null$0(ClientExecuteCommand.java:161) ~[seatunnel-starter.jar:2.3.3] at java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1640) ~[?:1.8.0_381] at java.util.concurrent.CompletableFuture$AsyncRun.exec(CompletableFuture.java:1632) ~[?:1.8.0_381] at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) ~[?:1.8.0_381] at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1067) ~[?:1.8.0_381] at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1703) ~[?:1.8.0_381] at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:172) ~[?:1.8.0_381] [INFO] 2023-10-23 16:40:47.449 +0800 - FINALIZE_SESSION ``` ### Zeta or Flink or Spark Version Zeta ### Java or Scala Version java version "1.8.0_381" ### Screenshots  ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
