Created JIRA for it: https://issues.apache.org/jira/browse/FLINK-17966

Regards
Bhaskar



On Wed, May 27, 2020 at 1:28 PM Vijay Bhaskar <bhaskar.eba...@gmail.com>
wrote:

> Thanks Yun. In that case  it would be good to give the reference of that
> documentation in the Flink Rest API:
> https://ci.apache.org/projects/flink/flink-docs-stable/monitoring/rest_api.html
> while explaining about the checkpoints. Tomorrow any one want to use REST
> API, they will get easy reference of the monitoring document of
> checkpoints. It would give them complete idea. So I will open Jira with
> this requirement
>
> Regards
> Bhaskar
>
> On Wed, May 27, 2020 at 11:59 AM Yun Tang <myas...@live.com> wrote:
>
>> To be honest, from my point of view current description should have
>> already give enough explanations [1] in "Overview Tab".
>> *    Latest Completed Checkpoint*: The latest successfully completed
>> checkpoints.
>>     *Latest Restore*: There are two types of restore operations.
>>
>>    - Restore from Checkpoint: We restored from a regular periodic
>>    checkpoint.
>>    - Restore from Savepoint: We restored from a savepoint.
>>
>>
>> You could still create a JIRA issue and give your ideas in that issue. If
>> agreed to work on in that ticket, you can create a PR to edit
>> checkpoint_monitoring.md [2] and checkpoint_monitoring.zh.md [3] to
>> update related documentation.
>>
>> [1]
>> https://ci.apache.org/projects/flink/flink-docs-release-1.10/monitoring/checkpoint_monitoring.html#overview-tab
>> [2]
>> https://github.com/apache/flink/blob/master/docs/monitoring/checkpoint_monitoring.md
>> [3]
>> https://github.com/apache/flink/blob/master/docs/monitoring/checkpoint_monitoring.zh.md
>>
>> Best
>> Yun Tang
>> ------------------------------
>> *From:* Vijay Bhaskar <bhaskar.eba...@gmail.com>
>> *Sent:* Tuesday, May 26, 2020 15:18
>> *To:* Yun Tang <myas...@live.com>
>> *Cc:* user <user@flink.apache.org>
>> *Subject:* Re: In consistent Check point API response
>>
>> Thanks Yun. How can i contribute better documentation of the same by
>> opening Jira on this?
>>
>> Regards
>> Bhaskar
>>
>> On Tue, May 26, 2020 at 12:32 PM Yun Tang <myas...@live.com> wrote:
>>
>> Hi Bhaskar
>>
>> I think I have understood your scenario now. And I think this is what
>> expected in Flink.
>> As you only allow your job could restore 5 times, the "restore" would
>> only record the checkpoint to restore at the 5th recovery, and the
>> checkpoint id would always stay there.
>>
>> "Restored" is for last restored checkpoint and "completed" is for last
>> completed checkpoint, they are actually not the same thing.
>> The only scenario that they're the same in numbers is when Flink just
>> restore successfully before a new checkpoint completes.
>>
>> Best
>> Yun Tang
>>
>>
>> ------------------------------
>> *From:* Vijay Bhaskar <bhaskar.eba...@gmail.com>
>> *Sent:* Tuesday, May 26, 2020 12:19
>> *To:* Yun Tang <myas...@live.com>
>> *Cc:* user <user@flink.apache.org>
>> *Subject:* Re: In consistent Check point API response
>>
>> Hi Yun
>> Understood the issue now:
>> "restored" always shows only the check point that is used for restoring
>> previous state
>> In all the attempts < 6 ( in my case max attempts are 5, 6 is the last
>> attempt)
>>   Flink HA is  restoring the state, so restored and latest are same value
>> if the last attempt  == 6
>>  Flink job already has few check points
>>  After that job failed and Flink HA gave up and marked the job state as
>> "FAILED"
>>    At this point "restored". value is the one which is in 5'th attempt
>> but latest is the one which is the latest checkpoint which is retained
>>
>> Shall i file any documentation improvement Jira? I want to add more
>> documentation with the help of  the above scenarios.
>>
>> Regards
>> Bhaskar
>>
>>
>>
>> On Tue, May 26, 2020 at 8:14 AM Yun Tang <myas...@live.com> wrote:
>>
>> Hi Bhaskar
>>
>> It seems I still not understand your case-5 totally. Your job failed 6
>> times, and recover from previous checkpoint to restart again. However, you
>> found the REST API told the wrong answer.
>> How do you ensure your "restored" field is giving the wrong checkpoint
>> file which is not latest? Have you ever checked the log in JM to view
>> related contents: "Restoring job xxx from latest valid checkpoint: x@xxxx"
>> [1] to know exactly which checkpoint choose to restore?
>>
>> I think you could give a more concrete example e.g. which expected/actual
>> checkpoint to restore, to tell your story.
>>
>> [1]
>> https://github.com/apache/flink/blob/8f992e8e868b846cf7fe8de23923358fc6b50721/flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointCoordinator.java#L1250
>>
>> Best
>> Yun Tang
>> ------------------------------
>> *From:* Vijay Bhaskar <bhaskar.eba...@gmail.com>
>> *Sent:* Monday, May 25, 2020 17:01
>> *To:* Yun Tang <myas...@live.com>
>> *Cc:* user <user@flink.apache.org>
>> *Subject:* Re: In consistent Check point API response
>>
>> Thanks Yun.
>> Here is the problem i am facing:
>>
>> I am using  jobs/:jobID/checkpoints  API to recover the failed job. We
>> have the remote manager which monitors the jobs.  We are using "restored"
>> field of the API response to get the latest check point file to use. Its
>> giving correct checkpoint file for all the 4 cases except the 5'th case.
>> Where the "restored" field is giving the wrong check point file which is
>> not latest.  When we compare the  check point file returned by  the
>> "completed". field, both are giving identical checkpoints in all 4 cases,
>> except 5'th case
>> We can't use flink UI in because of security reasons
>>
>> Regards
>> Bhaskar
>>
>> On Mon, May 25, 2020 at 12:57 PM Yun Tang <myas...@live.com> wrote:
>>
>> Hi Vijay
>>
>> If I understand correct, do you mean your last "restored" checkpoint is
>> null via REST api when the job failed 6 times and then recover successfully
>> with another several successful checkpoints?
>>
>> First of all, if your job just recovered successfully, can you observe
>> the "last restored" checkpoint in web UI?
>> Secondly, how long will you cannot see the "restored " field  after
>> recover successfully?
>> Last but not least, I cannot see the real difference among your cases,
>> what's the core difference in your case(5)?
>>
>> From the implementation of Flink, it will create the checkpoint statics
>> without restored checkpoint and assign it once the latest
>> savepoint/checkpoint is restored. [1]
>>
>> [1]
>> https://github.com/apache/flink/blob/50253c6b89e3c92cac23edda6556770a63643c90/flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointCoordinator.java#L1285
>>
>> Best
>> Yun Tang
>>
>> ------------------------------
>> *From:* Vijay Bhaskar <bhaskar.eba...@gmail.com>
>> *Sent:* Monday, May 25, 2020 14:20
>> *To:* user <user@flink.apache.org>
>> *Subject:* In consistent Check point API response
>>
>> Hi
>> I am using flink retained check points and along with
>>  jobs/:jobid/checkpoints API for retrieving the latest retained check point
>> Following the response of Flink Checkpoints API:
>>
>> I have my jobs restart attempts are 5
>>  check point API response in "latest" key, check point file name of both
>> "restored" and "completed" values are having following behavior
>> 1)Suppose the job is failed 3 times and recovered 4'th time, then both
>> values are same
>> 2)Suppose the job is failed 4 times and recovered 5'th time, then both
>> values are same
>> 3)Suppose the job is failed 5 times and recovered 6'th time, then both
>> values are same
>> 4) Suppose the job is failed all 6 times and the job marked failed. then
>> also both the values are same
>> 5)Suppose job is failed 6'th time , after recovering from 5 attempts
>> and made few check points, then both values are different.
>>
>> During case (1), case (2), case (3) and case (4) i never had any issue.
>> Only When case (5) i had severe issue in my production as the "restored "
>> field check point doesn't exist
>>
>> Please suggest any
>>
>>
>>
>> {
>>    "counts":{
>>       "restored":6,
>>       "total":3,
>>       "in_progress":0,
>>       "completed":3,
>>       "failed":0
>>    },
>>    "summary":{
>>       "state_size":{
>>          "min":4879,
>>          "max":4879,
>>          "avg":4879
>>       },
>>       "end_to_end_duration":{
>>          "min":25,
>>          "max":130,
>>          "avg":87
>>       },
>>       "alignment_buffered":{
>>          "min":0,
>>          "max":0,
>>          "avg":0
>>       }
>>    },
>>    "latest":{
>>       "completed":{
>>          "@class":"completed",
>>          "id":7094,
>>          "status":"COMPLETED",
>>          "is_savepoint":false,
>>          "trigger_timestamp":1590382502772,
>>          "latest_ack_timestamp":1590382502902,
>>          "state_size":4879,
>>          "end_to_end_duration":130,
>>          "alignment_buffered":0,
>>          "num_subtasks":2,
>>          "num_acknowledged_subtasks":2,
>>          "tasks":{
>>
>>          },
>>
>>  
>> "external_path":"file:/var/lib/persist/flink/checkpoints/29ae7600aa4f7d53a0dc1a0a7b257c85/chk-7094",
>>          "discarded":false
>>       },
>>       "savepoint":null,
>>       "failed":null,
>>       "restored":{
>>          "id":7093,
>>          "restore_timestamp":1590382478448,
>>          "is_savepoint":false,
>>
>>  
>> "external_path":"file:/var/lib/persist/flink/checkpoints/29ae7600aa4f7d53a0dc1a0a7b257c85/chk-7093"
>>       }
>>    },
>>    "history":[
>>       {
>>          "@class":"completed",
>>          "id":7094,
>>          "status":"COMPLETED",
>>          "is_savepoint":false,
>>          "trigger_timestamp":1590382502772,
>>          "latest_ack_timestamp":1590382502902,
>>          "state_size":4879,
>>          "end_to_end_duration":130,
>>          "alignment_buffered":0,
>>          "num_subtasks":2,
>>          "num_acknowledged_subtasks":2,
>>          "tasks":{
>>
>>          },
>>
>>  
>> "external_path":"file:/var/lib/persist/flink/checkpoints/29ae7600aa4f7d53a0dc1a0a7b257c85/chk-7094",
>>          "discarded":false
>>       },
>>       {
>>          "@class":"completed",
>>          "id":7093,
>>          "status":"COMPLETED",
>>          "is_savepoint":false,
>>          "trigger_timestamp":1590382310195,
>>          "latest_ack_timestamp":1590382310220,
>>          "state_size":4879,
>>          "end_to_end_duration":25,
>>          "alignment_buffered":0,
>>          "num_subtasks":2,
>>          "num_acknowledged_subtasks":2,
>>          "tasks":{
>>
>>          },
>>
>>  
>> "external_path":"file:/var/lib/persist/flink/checkpoints/29ae7600aa4f7d53a0dc1a0a7b257c85/chk-7093",
>>          "discarded":false
>>       },
>>       {
>>          "@class":"completed",
>>          "id":7092,
>>          "status":"COMPLETED",
>>          "is_savepoint":false,
>>          "trigger_timestamp":1590382190195,
>>          "latest_ack_timestamp":1590382190303,
>>          "state_size":4879,
>>          "end_to_end_duration":108,
>>          "alignment_buffered":0,
>>          "num_subtasks":2,
>>          "num_acknowledged_subtasks":2,
>>          "tasks":{
>>
>>          },
>>
>>  
>> "external_path":"file:/var/lib/persist/flink/checkpoints/29ae7600aa4f7d53a0dc1a0a7b257c85/chk-7092",
>>          "discarded":true
>>       }
>>    ]
>> }
>>
>>

Reply via email to