Hi, So, can't we extract all previous savepoint data by using ExistingSavepoint?
Thank you On Fri, Apr 30, 2021 at 10:25 AM David Anderson <dander...@apache.org> wrote: > Abdullah, > > The example you are studying -- the one using the state processor API -- > can be used with any retained checkpoint or savepoint created while running > the RidesAndFaresSolution job. But this is a very special use of > checkpoints and savepoints that shows how to extract data from them. > > Normally the state processor API is used with savepoints, and not with > checkpoints. This example uses checkpoints so that the example can be > easily run from the IDE, without requiring a local flink installation. > > The normal use for checkpoints is for failure recovery, while savepoints > are typically used for redeployments and rescaling -- and in these cases > the state processor API is not involved. You would use "flink run -s ..." > on the command line to manually resume from a checkpoint or savepoint, and > in the case of a job failure, the restart will happen automatically. > > The flink operations playground [1] is a great way to gain more > understanding of these aspects of flink. > > [1] > https://ci.apache.org/projects/flink/flink-docs-stable/try-flink/flink-operations-playground.html > > Best regards, > David > > On Fri, Apr 30, 2021 at 1:56 PM Abdullah bin Omar < > abdullahbinoma...@gmail.com> wrote: > >> Hi, >> >> Please answer me some of my below question whether my understanding >> correct or not, and please answer the direct ask questions. >> >> *Question no 1 (about dependency):* >> >> *What is dependency (in pom.xml) for the org.apache.flink.training?* >> >> I am trying to *import* >> org.apache.flink.training.exercises.common.sources.TaxiFareGenerator; >> However, it can not resolve. >> >> [note that, I am using the group id: <groupId>org.apache.flink</groupId> >> >> >> *Question No 2 (which one is being load to an existing savepoint):* >> >> >> According to my understanding after reading [1], the name >> "ExistingSavepoint" looks like that it will restore all previous savepoint. >> However, according to [2], the input file is only a checkpointed file. >> >> >> *(i)* *is that mean that we can only load the last checkpointed file (in >> case of job failure) by using the ExistingSavepoint to restart the job >> where it fails?* >> >> >> *(ii)* *and there is no option to load all previous savepoint. is this >> correct?* >> >> >> >> *Question No 3 (about loading an existing savepoint):* >> >> ExecutionEnvironment bEnv = ExecutionEnvironment. >> *getExecutionEnvironment*(); >> >> ExistingSavepoint sp = Savepoint.*load*(bEnv, "hdfs://path/", new >> MemoryStateBackend); >> >> >> >> This is the code for loading an existing savepoint. However, I configure >> a file location in flink conf to save the savepoint. So then, each time the >> job is running. I use a command in the terminal, ./bin/flink savepoint jobid >> >> and the savepointed file saved in the file location (that is set up in >> flink conf). >> >> >> In this case, to load the savepoint, file location will be the location >> that set up in the flink conf and FileSystemBackend will have to use >> instead of MemoryStateBackend. *is this correct?* >> >> >> >> >> [1] >> https://ci.apache.org/projects/flink/flink-docs-stable/dev/libs/state_processor_api.html >> >> [2] >> https://github.com/ververica/flink-training/blob/master/state-processor/src/main/java/com/ververica/flink/training/exercises/ReadRidesAndFaresSnapshot.java >> >> >> >> >> Thank you >> >> >> >> >> >> >> On Fri, Apr 23, 2021 at 10:10 AM David Anderson <dander...@apache.org> >> wrote: >> >>> Abdullah, >>> >>> ReadRidesAndFaresSnapshot [1] is an example that shows how to use the >>> State Processor API to display the contents of a snapshot taken while >>> running RidesAndFaresSolution [2]. >>> >>> Hopefully that will help you get started. >>> >>> [1] >>> https://github.com/ververica/flink-training/blob/master/state-processor/src/main/java/com/ververica/flink/training/exercises/ReadRidesAndFaresSnapshot.java >>> [2] >>> https://github.com/ververica/flink-training/blob/master/rides-and-fares/src/solution/java/org/apache/flink/training/solutions/ridesandfares/RidesAndFaresSolution.java >>> >>> Best regards, >>> David >>> >>> On Fri, Apr 23, 2021 at 3:32 PM Abdullah bin Omar < >>> abdullahbinoma...@gmail.com> wrote: >>> >>>> Hi, >>>> >>>> Thank you for your reply. >>>> >>>> I want to read the previous snapshot (if needed) at the time of >>>> operation. In [1], there is a portion: >>>> >>>> DataSet<Integer> listState = savepoint.readListState<>( >>>> "my-uid", >>>> "list-state", >>>> Types.INT); >>>> >>>> >>>> here, will the function savepoint.readliststate<> () work to read the >>>> previous snapshot? If it is, then is the filename of a savepoint file >>>> similar to my-uid? >>>> >>>> [1] >>>> https://ci.apache.org/projects/flink/flink-docs-stable/dev/libs/state_processor_api.html >>>> >>>> Thank you >>>> >>>> >>>> >>>> >>>> On Fri, Apr 23, 2021 at 1:11 AM Matthias Pohl <matth...@ververica.com> >>>> wrote: >>>> >>>>> What is it you're trying to achieve in general? The JavaDoc of >>>>> MetadataV2V3SerializerBase provides a description on the format of the >>>>> file. Theoretically, you could come up with custom code using the Flink >>>>> sources to parse the content of the file. But maybe, there's another way >>>>> to >>>>> accomplish what you're trying to do. >>>>> >>>>> Matthias >>>>> >>>>> [1] >>>>> https://github.com/apache/flink/blob/adaaed426c2e637b8e5ffa3f0d051326038d30aa/flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/metadata/MetadataV2V3SerializerBase.java#L83 >>>>> >>>>> On Thu, Apr 22, 2021 at 7:53 PM Abdullah bin Omar < >>>>> abdullahbinoma...@gmail.com> wrote: >>>>> >>>>>> Hi, >>>>>> >>>>>> I have a savepoint or checkpointed file from my task. However, the >>>>>> file is binary. I want to see what the file contains. >>>>>> >>>>>> How is it possible to see what information the file has (or how it is >>>>>> possible to make it human readable?) >>>>>> >>>>>> Thank you >>>>>> >>>>>> On Thu, Apr 22, 2021 at 10:19 AM Matthias Pohl < >>>>>> matth...@ververica.com> wrote: >>>>>> >>>>>>> Hi Abdullah, >>>>>>> the metadata file contains handles to the operator states of the >>>>>>> checkpoint [1]. You might want to have a look into the State Processor >>>>>>> API >>>>>>> [2]. >>>>>>> >>>>>>> Best, >>>>>>> Matthias >>>>>>> >>>>>>> [1] >>>>>>> https://github.com/apache/flink/blob/adaaed426c2e637b8e5ffa3f0d051326038d30aa/flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/metadata/MetadataV2V3SerializerBase.java#L83 >>>>>>> [2] >>>>>>> https://ci.apache.org/projects/flink/flink-docs-stable/dev/libs/state_processor_api.html >>>>>>> >>>>>>> On Thu, Apr 22, 2021 at 4:57 PM Abdullah bin Omar < >>>>>>> abdullahbinoma...@gmail.com> wrote: >>>>>>> >>>>>>>> Hi, >>>>>>>> >>>>>>>> (1) what 's the snapshot metadata file (binary) contains ? is it >>>>>>>> possible to read the snapshot metadata file by using Flink >>>>>>>> Deserialization? >>>>>>>> >>>>>>>> (2) is there any function that can be used to see the previous >>>>>>>> states on time of operation? >>>>>>>> >>>>>>>> Thank you >>>>>>>> >>>>>>> >>>>>
