> The job sub directory will be cleaned up when the job > finished/canceled/failed.
What does this mean? Also, to clarify: I'm a very sloppy developer. My jobs crash ALL the time... and yet, the jobs would ALWAYS resume from the last checkpoint. The only cases where I expect Flink to clean up the checkpoint data from ZK is when I explicitly stop or cancel the job (in those cases the job manager takes a savepoint before cleaning up zk and finishing the cluster). Which is not the case here. Flink was on autopilot here and decided to wipe my poor, good checkpoint metadata as the logs show. On Tue, Sep 8, 2020, at 7:59 PM, Yang Wang wrote: > AFAIK, the HA data, including Zookeeper meta data and real data on DFS, will > only be cleaned up > when the Flink cluster reached terminated state. > > So if you are using a session cluster, the root cluster node on Zk will be > cleaned up after you manually > stop the session cluster. The job sub directory will be cleaned up when the > job finished/canceled/failed. > > If you are using a job/application cluster, once the only running job > finished/failed, all the HA data will > be cleaned up. I think you need to check the job restart strategy you have > set. For example, the following > configuration will make the Flink cluster terminated after 10 attempts. > > restart-strategy: fixed-delay > restart-strategy.fixed-delay.attempts: 10 > > > Best, > Yang > > Cristian <k...@fastmail.fm> 于2020年9月9日周三 上午12:28写道： >> __ >> I'm using the standalone script to start the cluster. >> >> As far as I can tell, it's not easy to reproduce. We found that zookeeper >> lost a node around the time this happened, but all of our other 75 Flink >> jobs which use the same setup, version and zookeeper, didn't have any >> issues. They didn't even restart. >> >> So unfortunately I don't know how to reproduce this. All I know is I can't >> sleep. I have nightmares were my precious state is deleted. I wake up crying >> and quickly start manually savepointing all jobs just in case, because I >> feel the day of reckon is near. Flinkpocalypse! >> >> On Tue, Sep 8, 2020, at 5:54 AM, Robert Metzger wrote: >>> Thanks a lot for reporting this problem here Cristian! >>> >>> I am not super familiar with the involved components, but the behavior you >>> are describing doesn't sound right to me. >>> Which entrypoint are you using? This is logged at the beginning, like this: >>> "2020-09-08 14:45:32,807 INFO >>> org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Starting >>> StandaloneSessionClusterEntrypoint (Version: 1.11.1, Scala: 2.12, >>> Rev:7eb514a, Date:2020-07-15T07:02:09+02:00)" >>> >>> Do you know by chance if this problem is reproducible? With the >>> StandaloneSessionClusterEntrypoint I was not able to reproduce the problem. >>> >>> >>> >>> >>> On Tue, Sep 8, 2020 at 4:00 AM Husky Zeng <568793...@qq.com> wrote: >>>> Hi Cristian, >>>> >>>> >>>> I don't know if it was designed to be like this deliberately. >>>> >>>> So I have already submitted an issue ,and wait for somebody to response. >>>> >>>> https://issues.apache.org/jira/browse/FLINK-19154 >>>> >>>> >>>> >>>> -- >>>> Sent from: >>>> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/
