Just a bit more elaboration on the question that we need to answer here: Do we want to expose the internal ArchivedExecutionGraph data structure through JSON?
- The JSON approach allows the user to have (almost) full access to the information (that would be otherwise derived from the REST API). Therefore, there's no need to spin up a cluster to access this information. Any information that shall be exposed through the REST API needs to be well-defined in this JSON structure, though. Large parts of the ArchivedExecutionGraph data structure (essentially anything that shall be used to populate the REST API) become public domain, though, which puts more constraints on this data structure and makes it harder to change it in the future. - The binary data approach allows us to keep the data structure itself internal. We have more control over what we want to expose by providing access points in the ClusterClient (e.g. just add a command to extract the external storage path from the file). - The compromise (i.e. keeping ExecutionGraphInfoStore and JobResultStore separate and just expose the checkpoint information next to the JobResult in the JobResultStore file) would keep us the closest to the current state, requires the least code changes and the least exposure of internal data structures. It would allow any system (like the Kubernetes Operator) to extract the checkpoint's external storage path. But we would still be stuck with kind-of redundant components. >From a user's perspective, I feel like the JSON approach is the best one because it gives him/her the most freedom to be independent of Flink binaries when handling completed jobs. But I see benefits from a Flink developer's perspective to not expose the entire data structure but use the ClusterClient as an access point. The last option is my least favorite one: Moving the ExecutionGraphInfo out of the JobManager seems to be the right thing to do when thinking about Flink's vision to become cloud-native. Just my 2cts on that topic. Matthias On Mon, Sep 4, 2023 at 1:11 PM Matthias Pohl <matthias.p...@aiven.io> wrote: > Hi everyone, > I want to open the discussion on FLIP-360 [1]. The goal of this FLIP is to > combine the two very similar components ExecutionGraphInfoStore and > JobResultStore into a single component. > > The benefit of this effort would be to expose the metadata of a > globally-terminated job even in cases where the JobManager fails shortly > after the job finished. This is relevant for external checkpoint management > (like it's done in the Kubernetes Operator) which relies on the checkpoint > information to be available. > > More generally, it would allow completed jobs to be listed as part of the > Flink cluster even after a JM failover. This would allow users to gain more > control over finished jobs. > > The current state of the FLIP doesn't come up with a final conclusion on > the serialization format of the data (JSON vs binary). I want to emphasize > that there's also a third option which keeps both components separate and > only exposes the additional checkpoint information through the > JobResultStore. > > I'm looking forward to feedback. > Best, > Matthias > > PS: I might be less responsive in the next 2-3 weeks but want to initiate > the discussion, anyway. > > [1] > https://cwiki.apache.org/confluence/display/FLINK/FLIP-360%3A+Merging+the+ExecutionGraphInfoStore+and+the+JobResultStore+into+a+single+component+CompletedJobStore >