vertern created FLINK-39223:
-------------------------------
Summary: IllegalStateException: Failed to send request to
coordinator: SchemaChangeRequest
Key: FLINK-39223
URL: https://issues.apache.org/jira/browse/FLINK-39223
Project: Flink
Issue Type: Bug
Components: Flink CDC
Environment: flink 1.20.1
flink-cdc-3.5.0
Reporter: vertern
Exception
java.lang.IllegalStateException: Failed to send request to coordinator:
SchemaChangeRequest{tableId=*******}
at
org.apache.flink.cdc.runtime.operators.schema.regular.SchemaOperator.sendRequestToCoordinator(SchemaOperator.java:241)
at
org.apache.flink.cdc.runtime.operators.schema.regular.SchemaOperator.requestSchemaChange(SchemaOperator.java:227)
at
org.apache.flink.cdc.runtime.operators.schema.regular.SchemaOperator.handleSchemaChangeEvent(SchemaOperator.java:173)
at
org.apache.flink.cdc.runtime.operators.schema.regular.SchemaOperator.processElement(SchemaOperator.java:148)
at
org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.pushToOperator(CopyingChainingOutput.java:75)
at
org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:50)
at
org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:29)
at
org.apache.flink.streaming.runtime.tasks.SourceOperatorStreamTask$AsyncDataOutputToOutput.emitRecord(SourceOperatorStreamTask.java:310)
at
org.apache.flink.streaming.api.operators.source.SourceOutputWithWatermarks.collect(SourceOutputWithWatermarks.java:110)
at
org.apache.flink.streaming.api.operators.source.SourceOutputWithWatermarks.collect(SourceOutputWithWatermarks.java:101)
at
org.apache.flink.cdc.connectors.mysql.source.reader.MySqlPipelineRecordEmitter.processElement(MySqlPipelineRecordEmitter.java:151)
at
org.apache.flink.cdc.connectors.mysql.source.reader.MySqlRecordEmitter.emitRecord(MySqlRecordEmitter.java:74)
at
org.apache.flink.cdc.connectors.mysql.source.reader.MySqlRecordEmitter.emitRecord(MySqlRecordEmitter.java:47)
at
org.apache.flink.connector.base.source.reader.SourceReaderBase.pollNext(SourceReaderBase.java:203)
at
org.apache.flink.connector.base.source.reader.SourceReaderBase.pollNext(SourceReaderBase.java:216)
at
org.apache.flink.streaming.api.operators.SourceOperator.emitNext(SourceOperator.java:443)
at
org.apache.flink.streaming.runtime.io.StreamTaskSourceInput.emitNext(StreamTaskSourceInput.java:68)
at
org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:65)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:638)
at
org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:231)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:973)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:917)
at
org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:970)
at
org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:949)
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:763)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:575)
at java.base/java.lang.Thread.run(Unknown Source)
Caused by: java.util.concurrent.ExecutionException:
java.util.concurrent.TimeoutException: Invocation of
[RemoteRpcInvocation(JobMasterOperatorEventGateway.sendRequestToCoordinator(OperatorID,
SerializedValue))] at recipient
[pekko.tcp://flink@***:6123/user/rpc/jobmanager_8] timed out. This is usually
caused by: 1) Pekko failed sending the message silently, due to problems like
oversized payload or serialization failures. In that case, you should find
detailed error information in the logs. 2) The recipient needs more time for
responding, due to problems like slow machines or network jitters. In that
case, you can try to increase pekko.ask.timeout.
at java.base/java.util.concurrent.CompletableFuture.reportGet(Unknown
Source)
at java.base/java.util.concurrent.CompletableFuture.get(Unknown Source)
at
org.apache.flink.cdc.runtime.operators.schema.regular.SchemaOperator.sendRequestToCoordinator(SchemaOperator.java:238)
... 26 more
Caused by: java.util.concurrent.TimeoutException: Invocation of
[RemoteRpcInvocation(JobMasterOperatorEventGateway.sendRequestToCoordinator(OperatorID,
SerializedValue))] at recipient
[pekko.tcp://flink@***:6123/user/rpc/jobmanager_8] timed out. This is usually
caused by: 1) Pekko failed sending the message silently, due to problems like
oversized payload or serialization failures. In that case, you should find
detailed error information in the logs. 2) The recipient needs more time for
responding, due to problems like slow machines or network jitters. In that
case, you can try to increase pekko.ask.timeout.
at jdk.proxy3/jdk.proxy3.$Proxy38.sendRequestToCoordinator(Unknown
Source)
at
org.apache.flink.runtime.taskexecutor.rpc.RpcTaskOperatorEventGateway.sendRequestToCoordinator(RpcTaskOperatorEventGateway.java:73)
at
org.apache.flink.cdc.runtime.operators.schema.regular.SchemaOperator.sendRequestToCoordinator(SchemaOperator.java:235)
... 26 more
Caused by: org.apache.pekko.pattern.AskTimeoutException: Ask timed out on
[Actor[pekko.tcp://flink@***:6123/user/rpc/jobmanager_8#1929768345]] after
[30000 ms]. Message of type
[org.apache.flink.runtime.rpc.messages.RemoteFencedMessage]. A typical reason
for `AskTimeoutException` is that the recipient actor didn't send a reply.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
