Hello, We have added a serializer code which register all the schemas with executionEnvironment and is called before execute(). This is to make sure that all the avro schemas are pre-registered and cached in executors before actual execution begin. Now while submitting the job we are getting below error. executionEnvironment.addDefaultKryoSerializer(myKryo.class, largeSerializerObject);
Error : 13223 [flink-akka.actor.default-dispatcher-5] ERROR akka.remote.EndpointWriter - Transient association error (association remains live) akka.remote.OversizedPayloadException: Discarding oversized payload sent to Actor[akka.tcp://fl...@d191303-012.dc.gs.com:54840/user/jobmanager#1440921956]: max allowed size 10485760 bytes, actual size of encoded class org.apache.flink.runtime.messages.JobManagerMessages$LeaderSessionMessage was 45388747 bytes. We followed the documentation to increase akka.framesize to 200mb while starting yarn cluster but it still fails with above error. akka.framesize: Maximum size of messages which are sent between the JobManager and the TaskManagers. If Flink fails because messages exceed this limit, then you should increase it. The message size requires a size-unit specifier (DEFAULT: 10485760b). command to start yarn cluister: ./yarn-session.sh -n 15 -tm 24576 -jm 6144 -s 7 -Djobmanager.web.history=100 -Dakka.ask.timeout=10min -Dtaskmanager.network.numberOfBuffers=8000 -Dakka.framesize=209715200b We are using flink 1.2.0 version. Can you suggest if this is correct config to increase akka frame size? Or Are we using correct config to fix the issue? Thanks, Kasif
