Thanks Yun. How can i contribute better documentation of the same by opening Jira on this?
Regards Bhaskar On Tue, May 26, 2020 at 12:32 PM Yun Tang <myas...@live.com> wrote: > Hi Bhaskar > > I think I have understood your scenario now. And I think this is what > expected in Flink. > As you only allow your job could restore 5 times, the "restore" would only > record the checkpoint to restore at the 5th recovery, and the checkpoint id > would always stay there. > > "Restored" is for last restored checkpoint and "completed" is for last > completed checkpoint, they are actually not the same thing. > The only scenario that they're the same in numbers is when Flink just > restore successfully before a new checkpoint completes. > > Best > Yun Tang > > > ------------------------------ > *From:* Vijay Bhaskar <bhaskar.eba...@gmail.com> > *Sent:* Tuesday, May 26, 2020 12:19 > *To:* Yun Tang <myas...@live.com> > *Cc:* user <user@flink.apache.org> > *Subject:* Re: In consistent Check point API response > > Hi Yun > Understood the issue now: > "restored" always shows only the check point that is used for restoring > previous state > In all the attempts < 6 ( in my case max attempts are 5, 6 is the last > attempt) > Flink HA is restoring the state, so restored and latest are same value > if the last attempt == 6 > Flink job already has few check points > After that job failed and Flink HA gave up and marked the job state as > "FAILED" > At this point "restored". value is the one which is in 5'th attempt but > latest is the one which is the latest checkpoint which is retained > > Shall i file any documentation improvement Jira? I want to add more > documentation with the help of the above scenarios. > > Regards > Bhaskar > > > > On Tue, May 26, 2020 at 8:14 AM Yun Tang <myas...@live.com> wrote: > > Hi Bhaskar > > It seems I still not understand your case-5 totally. Your job failed 6 > times, and recover from previous checkpoint to restart again. However, you > found the REST API told the wrong answer. > How do you ensure your "restored" field is giving the wrong checkpoint > file which is not latest? Have you ever checked the log in JM to view > related contents: "Restoring job xxx from latest valid checkpoint: x@xxxx" > [1] to know exactly which checkpoint choose to restore? > > I think you could give a more concrete example e.g. which expected/actual > checkpoint to restore, to tell your story. > > [1] > https://github.com/apache/flink/blob/8f992e8e868b846cf7fe8de23923358fc6b50721/flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointCoordinator.java#L1250 > > Best > Yun Tang > ------------------------------ > *From:* Vijay Bhaskar <bhaskar.eba...@gmail.com> > *Sent:* Monday, May 25, 2020 17:01 > *To:* Yun Tang <myas...@live.com> > *Cc:* user <user@flink.apache.org> > *Subject:* Re: In consistent Check point API response > > Thanks Yun. > Here is the problem i am facing: > > I am using jobs/:jobID/checkpoints API to recover the failed job. We > have the remote manager which monitors the jobs. We are using "restored" > field of the API response to get the latest check point file to use. Its > giving correct checkpoint file for all the 4 cases except the 5'th case. > Where the "restored" field is giving the wrong check point file which is > not latest. When we compare the check point file returned by the > "completed". field, both are giving identical checkpoints in all 4 cases, > except 5'th case > We can't use flink UI in because of security reasons > > Regards > Bhaskar > > On Mon, May 25, 2020 at 12:57 PM Yun Tang <myas...@live.com> wrote: > > Hi Vijay > > If I understand correct, do you mean your last "restored" checkpoint is > null via REST api when the job failed 6 times and then recover successfully > with another several successful checkpoints? > > First of all, if your job just recovered successfully, can you observe the > "last restored" checkpoint in web UI? > Secondly, how long will you cannot see the "restored " field after > recover successfully? > Last but not least, I cannot see the real difference among your cases, > what's the core difference in your case(5)? > > From the implementation of Flink, it will create the checkpoint statics > without restored checkpoint and assign it once the latest > savepoint/checkpoint is restored. [1] > > [1] > https://github.com/apache/flink/blob/50253c6b89e3c92cac23edda6556770a63643c90/flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointCoordinator.java#L1285 > > Best > Yun Tang > > ------------------------------ > *From:* Vijay Bhaskar <bhaskar.eba...@gmail.com> > *Sent:* Monday, May 25, 2020 14:20 > *To:* user <user@flink.apache.org> > *Subject:* In consistent Check point API response > > Hi > I am using flink retained check points and along with > jobs/:jobid/checkpoints API for retrieving the latest retained check point > Following the response of Flink Checkpoints API: > > I have my jobs restart attempts are 5 > check point API response in "latest" key, check point file name of both > "restored" and "completed" values are having following behavior > 1)Suppose the job is failed 3 times and recovered 4'th time, then both > values are same > 2)Suppose the job is failed 4 times and recovered 5'th time, then both > values are same > 3)Suppose the job is failed 5 times and recovered 6'th time, then both > values are same > 4) Suppose the job is failed all 6 times and the job marked failed. then > also both the values are same > 5)Suppose job is failed 6'th time , after recovering from 5 attempts > and made few check points, then both values are different. > > During case (1), case (2), case (3) and case (4) i never had any issue. > Only When case (5) i had severe issue in my production as the "restored " > field check point doesn't exist > > Please suggest any > > > > { > "counts":{ > "restored":6, > "total":3, > "in_progress":0, > "completed":3, > "failed":0 > }, > "summary":{ > "state_size":{ > "min":4879, > "max":4879, > "avg":4879 > }, > "end_to_end_duration":{ > "min":25, > "max":130, > "avg":87 > }, > "alignment_buffered":{ > "min":0, > "max":0, > "avg":0 > } > }, > "latest":{ > "completed":{ > "@class":"completed", > "id":7094, > "status":"COMPLETED", > "is_savepoint":false, > "trigger_timestamp":1590382502772, > "latest_ack_timestamp":1590382502902, > "state_size":4879, > "end_to_end_duration":130, > "alignment_buffered":0, > "num_subtasks":2, > "num_acknowledged_subtasks":2, > "tasks":{ > > }, > > > "external_path":"file:/var/lib/persist/flink/checkpoints/29ae7600aa4f7d53a0dc1a0a7b257c85/chk-7094", > "discarded":false > }, > "savepoint":null, > "failed":null, > "restored":{ > "id":7093, > "restore_timestamp":1590382478448, > "is_savepoint":false, > > > "external_path":"file:/var/lib/persist/flink/checkpoints/29ae7600aa4f7d53a0dc1a0a7b257c85/chk-7093" > } > }, > "history":[ > { > "@class":"completed", > "id":7094, > "status":"COMPLETED", > "is_savepoint":false, > "trigger_timestamp":1590382502772, > "latest_ack_timestamp":1590382502902, > "state_size":4879, > "end_to_end_duration":130, > "alignment_buffered":0, > "num_subtasks":2, > "num_acknowledged_subtasks":2, > "tasks":{ > > }, > > > "external_path":"file:/var/lib/persist/flink/checkpoints/29ae7600aa4f7d53a0dc1a0a7b257c85/chk-7094", > "discarded":false > }, > { > "@class":"completed", > "id":7093, > "status":"COMPLETED", > "is_savepoint":false, > "trigger_timestamp":1590382310195, > "latest_ack_timestamp":1590382310220, > "state_size":4879, > "end_to_end_duration":25, > "alignment_buffered":0, > "num_subtasks":2, > "num_acknowledged_subtasks":2, > "tasks":{ > > }, > > > "external_path":"file:/var/lib/persist/flink/checkpoints/29ae7600aa4f7d53a0dc1a0a7b257c85/chk-7093", > "discarded":false > }, > { > "@class":"completed", > "id":7092, > "status":"COMPLETED", > "is_savepoint":false, > "trigger_timestamp":1590382190195, > "latest_ack_timestamp":1590382190303, > "state_size":4879, > "end_to_end_duration":108, > "alignment_buffered":0, > "num_subtasks":2, > "num_acknowledged_subtasks":2, > "tasks":{ > > }, > > > "external_path":"file:/var/lib/persist/flink/checkpoints/29ae7600aa4f7d53a0dc1a0a7b257c85/chk-7092", > "discarded":true > } > ] > } > >
