Hi, Please answer me some of my below question whether my understanding correct or not, and please answer the direct ask questions.
*Question no 1 (about dependency):* *What is dependency (in pom.xml) for the org.apache.flink.training?* I am trying to *import* org.apache.flink.training.exercises.common.sources.TaxiFareGenerator; However, it can not resolve. [note that, I am using the group id: <groupId>org.apache.flink</groupId> *Question No 2 (which one is being load to an existing savepoint):* According to my understanding after reading [1], the name "ExistingSavepoint" looks like that it will restore all previous savepoint. However, according to [2], the input file is only a checkpointed file. *(i)* *is that mean that we can only load the last checkpointed file (in case of job failure) by using the ExistingSavepoint to restart the job where it fails?* *(ii)* *and there is no option to load all previous savepoint. is this correct?* *Question No 3 (about loading an existing savepoint):* ExecutionEnvironment bEnv = ExecutionEnvironment.*getExecutionEnvironment* (); ExistingSavepoint sp = Savepoint.*load*(bEnv, "hdfs://path/", new MemoryStateBackend); This is the code for loading an existing savepoint. However, I configure a file location in flink conf to save the savepoint. So then, each time the job is running. I use a command in the terminal, ./bin/flink savepoint jobid and the savepointed file saved in the file location (that is set up in flink conf). In this case, to load the savepoint, file location will be the location that set up in the flink conf and FileSystemBackend will have to use instead of MemoryStateBackend. *is this correct?* [1] https://ci.apache.org/projects/flink/flink-docs-stable/dev/libs/state_processor_api.html [2] https://github.com/ververica/flink-training/blob/master/state-processor/src/main/java/com/ververica/flink/training/exercises/ReadRidesAndFaresSnapshot.java Thank you On Fri, Apr 23, 2021 at 10:10 AM David Anderson <dander...@apache.org> wrote: > Abdullah, > > ReadRidesAndFaresSnapshot [1] is an example that shows how to use the > State Processor API to display the contents of a snapshot taken while > running RidesAndFaresSolution [2]. > > Hopefully that will help you get started. > > [1] > https://github.com/ververica/flink-training/blob/master/state-processor/src/main/java/com/ververica/flink/training/exercises/ReadRidesAndFaresSnapshot.java > [2] > https://github.com/ververica/flink-training/blob/master/rides-and-fares/src/solution/java/org/apache/flink/training/solutions/ridesandfares/RidesAndFaresSolution.java > > Best regards, > David > > On Fri, Apr 23, 2021 at 3:32 PM Abdullah bin Omar < > abdullahbinoma...@gmail.com> wrote: > >> Hi, >> >> Thank you for your reply. >> >> I want to read the previous snapshot (if needed) at the time of >> operation. In [1], there is a portion: >> >> DataSet<Integer> listState = savepoint.readListState<>( >> "my-uid", >> "list-state", >> Types.INT); >> >> >> here, will the function savepoint.readliststate<> () work to read the >> previous snapshot? If it is, then is the filename of a savepoint file >> similar to my-uid? >> >> [1] >> https://ci.apache.org/projects/flink/flink-docs-stable/dev/libs/state_processor_api.html >> >> Thank you >> >> >> >> >> On Fri, Apr 23, 2021 at 1:11 AM Matthias Pohl <matth...@ververica.com> >> wrote: >> >>> What is it you're trying to achieve in general? The JavaDoc of >>> MetadataV2V3SerializerBase provides a description on the format of the >>> file. Theoretically, you could come up with custom code using the Flink >>> sources to parse the content of the file. But maybe, there's another way to >>> accomplish what you're trying to do. >>> >>> Matthias >>> >>> [1] >>> https://github.com/apache/flink/blob/adaaed426c2e637b8e5ffa3f0d051326038d30aa/flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/metadata/MetadataV2V3SerializerBase.java#L83 >>> >>> On Thu, Apr 22, 2021 at 7:53 PM Abdullah bin Omar < >>> abdullahbinoma...@gmail.com> wrote: >>> >>>> Hi, >>>> >>>> I have a savepoint or checkpointed file from my task. However, the file >>>> is binary. I want to see what the file contains. >>>> >>>> How is it possible to see what information the file has (or how it is >>>> possible to make it human readable?) >>>> >>>> Thank you >>>> >>>> On Thu, Apr 22, 2021 at 10:19 AM Matthias Pohl <matth...@ververica.com> >>>> wrote: >>>> >>>>> Hi Abdullah, >>>>> the metadata file contains handles to the operator states of the >>>>> checkpoint [1]. You might want to have a look into the State Processor API >>>>> [2]. >>>>> >>>>> Best, >>>>> Matthias >>>>> >>>>> [1] >>>>> https://github.com/apache/flink/blob/adaaed426c2e637b8e5ffa3f0d051326038d30aa/flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/metadata/MetadataV2V3SerializerBase.java#L83 >>>>> [2] >>>>> https://ci.apache.org/projects/flink/flink-docs-stable/dev/libs/state_processor_api.html >>>>> >>>>> On Thu, Apr 22, 2021 at 4:57 PM Abdullah bin Omar < >>>>> abdullahbinoma...@gmail.com> wrote: >>>>> >>>>>> Hi, >>>>>> >>>>>> (1) what 's the snapshot metadata file (binary) contains ? is it >>>>>> possible to read the snapshot metadata file by using Flink >>>>>> Deserialization? >>>>>> >>>>>> (2) is there any function that can be used to see the previous >>>>>> states on time of operation? >>>>>> >>>>>> Thank you >>>>>> >>>>> >>>
