Hi team, Thank you for the proposal, Gabor and the review, Zakelly. Let me provide more context on the intended outcome, let us agree on the desirability of that first and then we can come back to the technical details.
The pain point that we observed with multiple users is that their operations teams cannot effectively read the Flink state. These folks are usually not Java developers (unlike the team who wrote the application in the first place), so we cannot reasonably expect them to be able to efficiently use the State Processor API, especially not in high pressure situations. Providing a FlinkSQL interface for these folks would be ideal, especially over specific checkpoints/savepoints. Spark has recently introduced this feature [1] and we have users who are considering switching to Spark as a result. You can consider the motivation to be a better alternative to the deprecated queryable state effort, which was never fully embraced, and rightfully so it lacked security and put unnecessary load on the live Flink application. I like Gabor's intended approach much more. What do you think? [1] https://www.databricks.com/blog/announcing-state-reader-api-new-statestore-data-source On Thu, Aug 8, 2024 at 12:06 PM Zakelly Lan <zakelly....@gmail.com> wrote: > Hi Gabor, > > Thanks for the proposal! However, I find it a little strange. Are you > saying the user can set the operator uid but then doesn't know what they > set when debugging? Otherwise, is the `OperatorIdentifier.forUid("my-uid")` > feasible? I understand your point about potential cross-team work, but the > person may not be able to debug code that was not written by them. Things > get complex in this scenario. Could you provide more details about the > issue you are facing? > > Regarding the checkpoint, it is not designed to be self-contained or > human-readable. I suggest not introducing such columns for debugging > purposes. > > > Best, > Zakelly > > On Wed, Aug 7, 2024 at 10:07 PM Gabor Somogyi <gabor.g.somo...@gmail.com> > wrote: > > > Hi Devs, > > > > I would like to start a discussion on FLIP-474: Store operator name and > UID > > in state metadata[1]. > > > > In short users are interested in what kind of operators are inside a > > checkpoint data which can be enhanced from user experience perspective. > The > > details can be found in FLIP-474[1]. > > > > Please share your thoughts on this. > > > > [1] > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-474%3A+Store+operator+name+and+UID+in+state+metadata > > > > BR, > > G > > >
