Hi Gary: From 1.5/1.6 document:
Configuring task-local recovery Task-local recovery is deactivated by default and can be activated through Flink’s configuration with the key state.backend.local-recovery as specified in CheckpointingOptions.LOCAL_RECOVERY. The value for this setting can either be true to enable or false(default) to disable local recovery. By default, local recovery is deactive. In 1.5.0, I’ve not enable local recovery. So whether I need manual disable local recovery via flink.conf? Regards James From: "James (Jian Wu) [FDS Data Platform]" <james...@coupang.com> Date: Monday, September 3, 2018 at 4:13 PM To: Gary Yao <g...@data-artisans.com> Cc: "user@flink.apache.org" <user@flink.apache.org> Subject: Re: Flink on Yarn, restart job will not destroy original task manager My Flink version is 1.5, I will rebuild new version flink Regards James From: Gary Yao <g...@data-artisans.com> Date: Monday, September 3, 2018 at 3:57 PM To: "James (Jian Wu) [FDS Data Platform]" <james...@coupang.com> Cc: "user@flink.apache.org" <user@flink.apache.org> Subject: Re: Flink on Yarn, restart job will not destroy original task manager Hi James, What version of Flink are you running? In 1.5.0, tasks can spread out due to changes that were introduced to support "local recovery". There is a mitigation in 1.5.1 that prevents task spread out but local recovery must be disabled [2]. Best, Gary [1] https://issues.apache.org/jira/browse/FLINK-9635 [2] https://issues.apache.org/jira/browse/FLINK-9634 On Mon, Sep 3, 2018 at 9:20 AM, James (Jian Wu) [FDS Data Platform] <james...@coupang.com<mailto:james...@coupang.com>> wrote: Hi: I launch flink application on yarn with 5 task manager, every task manager has 5 slots with such script #!/bin/sh CLASSNAME=$1 JARNAME=$2 ARUGMENTS=$3 export JVM_ARGS="${JVM_ARGS} -Dmill.env.active=aws" /usr/bin/flink run -m yarn-cluster --parallelism 15 -yn 5 -ys 3 -yjm 8192 -ytm 8192 -ynm flink-order-detection -yD env.java.opts.jobmanager='-Dmill.env.active=aws' -yD env.java.opts.taskmanager='-Dmill.env.active=aws' -c $CLASSNAME \ $JARNAME $ARUGMENTS The original flink app occupy 5 containers and 15 vcores, run for 3+ days, one of task manage killed by yarn because of memory leak and job manager start new task managers. Currently my flink app running normally on yarn, but occupy 10 containers, 28 vcores. (Application Master shows my flink job running for 75 hours, click into running job in flink web ui, it shows my job running for 28hours because of restart) In my opinion, job manager will attempt to start the failed task manager, and in the final app still use 5 containers and 15 vcores, why after restart job by yarn will occupy double resource. Any one can give me some suggestion? Regards James
