Hi Zakelly, Thanks for the feedback, let me elaborate on this.
In short Databricks has created a much more user friendly solution[1] for state observability (based on Flink's state processor API) than what we have now. Up until now our state processor API was good enough but now we're lagging behind. We see users (just like Spark) where the first class citizen is the state itself and they're pointing to the new Spark solution. Since the state became first class citizen there is a natural need to use it for business logic validation, debugging, explanatory browsing, etc... The main message here is that there are cases where users are not able to identify operators because hash is a one way conversion. I'm open to any suggestion but somehow the initial operator human readable identifier must be available. Let me come up with examples where users are completely blind. > Are you saying the user can set the operator uid but then doesn't know what they set when debugging? There are cases where the user is setting the UID in the job, such case it's not user friendly to parse git repos but doable. But there are cases where the user has limited or no control related UIDs: * SQL jobs are generating operators with meaningful names, but I think it's not realistic to enforce users to understand all the internals of Flink SQL implementation (which operator named where and how). * Iceberg is using the given UID as prefix and generating more operators with it * Weak justification but exists: Since operator name and UID are both optional some of the users are setting name only. Such case Flink generates a random hash, where only name can give some pointers. Hope I've given better context. [1] https://www.databricks.com/blog/announcing-state-reader-api-new-statestore-data-source BR, G On Thu, Aug 8, 2024 at 12:06 PM Zakelly Lan <zakelly....@gmail.com> wrote: > Hi Gabor, > > Thanks for the proposal! However, I find it a little strange. Are you > saying the user can set the operator uid but then doesn't know what they > set when debugging? Otherwise, is the `OperatorIdentifier.forUid("my-uid")` > feasible? I understand your point about potential cross-team work, but the > person may not be able to debug code that was not written by them. Things > get complex in this scenario. Could you provide more details about the > issue you are facing? > > Regarding the checkpoint, it is not designed to be self-contained or > human-readable. I suggest not introducing such columns for debugging > purposes. > > > Best, > Zakelly > > On Wed, Aug 7, 2024 at 10:07 PM Gabor Somogyi <gabor.g.somo...@gmail.com> > wrote: > > > Hi Devs, > > > > I would like to start a discussion on FLIP-474: Store operator name and > UID > > in state metadata[1]. > > > > In short users are interested in what kind of operators are inside a > > checkpoint data which can be enhanced from user experience perspective. > The > > details can be found in FLIP-474[1]. > > > > Please share your thoughts on this. > > > > [1] > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-474%3A+Store+operator+name+and+UID+in+state+metadata > > > > BR, > > G > > >
