Hi Yuval, I'm also wondering why do you have such a big metadata file. Probably, you could reduce it by decreasing "state.backend.fs.memory-threshold" (if you didn't do so already) [1].
[1] https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/state/checkpointing.html#state-backend-fs-memory-threshold Regards, Roman On Mon, Oct 19, 2020 at 7:26 AM Yun Tang <myas...@live.com> wrote: > Hi Yuval > > First of all, large savepoint metadata would not must need a very large > akka frame size. Writing meta data to external file system calls IO-write > method [1] instead of sending RPC message. > > Secondly, savepoint would not store any confiuration, it would only store > checkpointed state. > > BTW, why you could have so large RPC message over than 1GB? > > [1] > https://github.com/apache/flink/blob/f705f0af6ba50f6e68c22484d1daeda842518d27/flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/PendingCheckpoint.java#L313 > > Best > Yun Tang > ------------------------------ > *From:* Yuval Itzchakov <yuva...@gmail.com> > *Sent:* Thursday, October 15, 2020 21:22 > *To:* user <user@flink.apache.org> > *Subject:* akka.framesize configuration does not runtime execution > > Hi, > > Due to a very large savepoint metadata file (3GB +), I've set the > akka.framesize that is being required to 5GB. I set this via flink.conf > `akka.framesize` property. > > When trying to recover from the savepoint, the JM emits the following > error message: > > "thread":"flink-akka.actor.default-dispatcher-30" > "level":"ERROR" > "loggerName":"akka.remote.EndpointWriter" > "message":"Transient "Discarding oversized payload sent to > Actor[akka.tcp://flink@XXX:XXX/user/taskmanager_0#369979612]: max allowed > size 1073741824 bytes, actual size of encoded class > org.apache.flink.runtime.rpc.messages.RemoteRpcInvocation was 1610683118 > "name":"akka.remote.OversizedPayloadException" > > As I recall, while taking the savepoint the maximum framesize was indeed > defined as 1GB. > > Could it be that akka.framesize is being recovered from the stored > savepoint, thus not allowing me to configure re-configure the maximum size > of the payload? > > -- > Best Regards, > Yuval Itzchakov. >
