Hi, We are running a Flink 1.19 application under heavy load. and out job manager failed to restart with exceptions like
"Transient association error (association remains live)org.apache.pekko.remote.OversizedPayloadException: Discarding oversized payload sent to Actor[pekko.ssl.tcp://fl...@aiops-ir-lifecycle-eventprocessor-ep-taskmanager-1.aiops-ir-lifecycle-eventprocessor-ep-taskmanager.cp4waiops.svc:6122/user/rpc/taskmanager_0#2078073862]: max allowed size 10485760 bytes, actual size of encoded class org.apache.flink.runtime.rpc.messages.RemoteRpcInvocation was 13937663 bytes." and " Java.lang.Exception: Cannot deploy task policy-exec::schedule-policy-execution -> ... blah blah blah This is usually caused by: 1) Pekko failed sending the message silently, due to problems like oversized payload or serialization failures. In that case, you should find detailed error information in the logs. 2) The recipient needs more time for responding, due to problems like slow machines or network jitters. In that case, you can try to increase pekko.ask.timeout. at org.apache.flink.runtime.jobmaster.RpcTaskManagerGateway.submitTask(RpcTaskManagerGateway.java:59) ... at java.base/java.lang.Thread.run(Thread.java:857) Caused by: org.apache.pekko.pattern.AskTimeoutException: Ask timed out on [Actor[pekko.ssl.tcp://flink@]] after [10000 ms]. Message of type [org.apache.flink.runtime.rpc.messages.RemoteRpcInvocation]. A typical reason for `AskTimeoutException` is that the recipient actor didn't send a reply. " We saw there is a property pekko.framesize and we think what will solve our issue. But we'd like to understand the meaning behind this. Is that a limit on the size of the payload send from the jobmanager to the various taskmanager, and that default limit is 10MB ? Q: where do the data driving that limit comes from? whta can we do in our application to control this? Thanks JM Unless otherwise stated above: IBM United Kingdom Limited Registered in England and Wales with number 741598 Registered office: Building C, IBM Hursley Office, Hursley Park Road, Winchester, Hampshire SO21 2JN
