Hi All, Based on our agreement I've created a draft Flink state observability umbrella [1]. Please share your comments. It contains some details to give some insights but the focus would be on the direction.
[1] https://docs.google.com/document/d/1Du1-TShoOjaNDCahs3sgLWIpYkXzJPdSkgHcWLpELyw/edit BR, G On Sat, Aug 10, 2024 at 10:54 AM Zakelly Lan <zakelly....@gmail.com> wrote: > Hi Gabor, > > I apologize for any confusion. Let me clarify my position. > > The concept of state observability is important for users, and the current > FLIP seems to be a step in the right direction. However, before we proceed, > I suggest we discuss the final presentation of the state observability to > the user and consider the high-level vision for achieving this. It's > essential to ensure that the current FLIP aligns with the overall > objective. I'm not suggesting a comprehensive FLIP to address all the > missing pieces, and one FLIP for each piece is fine for me. I just want to > ensure that we are on the same page in terms of vision. The last thing I > want is a fragmented approach resulting in refactoring or deprecation of > code when we need a complete feature. > > Actually, I would hesitate about the current proposal of adding uid *in* > the state metadata. It may cause state incompatibility issues across > versions. In theory we can do this but it is better not if we are adding > data not for fault tolerance but only for human readability. And it could > be worse if we add one or two columns sporadically in future. > > In fact, I expect the state metadata store to exist next to the checkpoint > metadata, rather than within it. This gives us enough flexibility to polish > this function as users need it, and without breaking checkpoint > compatibility too often. Or moreover we don't have to stick to the form of > checkpoint and we could choose a more human readable format like json for > the metadata store. This is where I think this FLIP is inconsistent with my > expectation of the state observability approach. These considerations > deserve a discussion before proceeding with other details. WDTY? > > > Best, > Zakelly > > On Fri, Aug 9, 2024 at 8:22 PM Gabor Somogyi <gabor.g.somo...@gmail.com> > wrote: > > > Hi David, > > > > Thanks for sharing your thoughts! > > > > > It sounds like you might already have an end-to-end solution in mind. > It > > would be really helpful if you could put that into writing so we can all > > align our thinking. > > > > It makes sense to create a high level vision. > > > > > I’m not a fan of the mindset of “this is how it was done in Spark, so > > we’ll > > just replicate it” without proper discussion. We’ve had similar > > conversations before. > > > > I think we've had this conversation already in case of delegation token > > framework > > and I can say the same. No intention to take over things blindly but it's > > not a shame > > to be inspired by solutions which are welcome by users. > > The intention is similar just like in scalable authentication area where > > Flink is now ahead of Spark. > > > > > Would it be too much to ask for a FLIP that outlines the overall vision > > (without delving too deeply into the details) to ensure everyone is > aligned > > and moving in the same direction? > > > > That's a fair point and a constructive way how we can proceed. > > I'm going to come back with the details... > > > > BR, > > G > > > > > > On Fri, Aug 9, 2024 at 1:36 PM David Morávek <d...@apache.org> wrote: > > > > > Hi Gabor, > > > > > > Thanks for taking the initiative on this. It’s clear that significant > > > improvements are needed in this area, and parsing state files can be > > > incredibly challenging, even for those who are well-versed in it. > > > > > > > Just to make it crystal clear, I’m not shooting for an ad-hoc tiny > fix > > > but started a path where we fill each and every gap which will end up > in > > a > > > functionality and UX bar just like the Spark solution. > > > > > > It sounds like you might already have an end-to-end solution in mind. > It > > > would be really helpful if you could put that into writing so we can > all > > > align our thinking. > > > > > > I’m not a fan of the mindset of “this is how it was done in Spark, so > > we’ll > > > just replicate it” without proper discussion. We’ve had similar > > > conversations before. > > > > > > > But this doesn’t mean we create a single giga big FLIP after several > > > months of discussion. > > > > > > I don’t think anyone is asking for a massive FLIP after lengthy > > > discussions, but having a document that outlines the overall vision > could > > > be incredibly valuable, especially in a distributed setting. It also > > opens > > > the door for others to contribute to and shape this shared vision, > which > > is > > > a core principle of community-driven open-source development. > > > > > > Would it be too much to ask for a FLIP that outlines the overall vision > > > (without delving too deeply into the details) to ensure everyone is > > aligned > > > and moving in the same direction? > > > > > > Best, > > > D. > > > > > > On Fri, Aug 9, 2024 at 11:44 AM Gabor Somogyi < > gabor.g.somo...@gmail.com > > > > > > wrote: > > > > > > > Hi Zakelly, > > > > > > > > > I'd suggest we could think of this as a whole > > > > > > > > In general I think we have the same idea in our mind about > considering > > > the > > > > state observability as a whole, just we need to agree about the > > physical > > > > task scheduling. > > > > > > > > > But such a solution requires more design and discussion > > > > > > > > I can't even agree more. But this doesn't mean we create a single > giga > > > big > > > > FLIP after several months of discussion. > > > > > > > > > Regarding the current issue you are facing, here's my idea > > > > > > > > Just to make it crystal clear, I'm not shooting for ad-hoc tiny fix > but > > > > started a path where we fill each and every gap which will end-up in > a > > > > solution > > > > where we hit the functionality and UX bar just like the Spark > solution. > > > > From plan and code perspective I'm more ahead of this FLIP. > > > > So when you aim for different task scheduling then make your exact > > > > suggestion instead of providing hacks. > > > > > > > > If I assume correctly you suggest to create a FLIP where we define > and > > > > agree all the missing pieces in a single giga big FLIP, right? > > > > I would say there are obvious missing pieces which are clear that > they > > > > needed. Just like in PRs the more consumable pieces we have > > > > the better because this single change is about 1k lines of code. > Having > > > an > > > > overkill FLIP/PR can end up in feature creep which I think > > > > is disadvantageous. > > > > > > > > Of course this doesn't exclude the possibility that we start general > > more > > > > high level discussion about the whole state observability story. > > > > Here are my high level conceptual points (I consider roughly each > point > > > as > > > > a separate FLIP): > > > > * Store human readable IDs for operators in metadata > > > > * Expose the metadata as data stream > > > > * Store state with user defined schemas as self containing entity > > > > * SQL integration > > > > * State metastore with all the created checkpoints/savepoints > > > > * State file cleanup strategy in case of failure > > > > * Optional: Some extra tool like metadata explorer > > > > > > > > That said I suggest to split the higher level discussion from this > FLIP > > > in > > > > a separate thread. > > > > > > > > BR, > > > > G > > > > > > > > > > > > On Fri, Aug 9, 2024 at 10:17 AM Zakelly Lan <zakelly....@gmail.com> > > > wrote: > > > > > > > > > Hi Márton and Gabor, > > > > > > > > > > Thanks for sharing context! > > > > > > > > > > Yes, I'd admit that users need a more friendly way to explore > states. > > > And > > > > > it seems Flink lacks something like the state metadata store. I'd > > > suggest > > > > > we could think of this as a whole, to store enough information for > > > > > querying, including operator names, uids, hashes, as well as the > > state > > > > > types or descriptors. Moreover we provide a tool to list those > > > metadata. > > > > My > > > > > thoughts is to provide a complete solution instead of adding one or > > two > > > > > specific data alongside the checkpoint. WDTY? I believe with the > > state > > > > > schema queryable, the State Processor API could become more > powerful > > > and > > > > > easier to use. > > > > > > > > > > But such a solution requires more design and discussion. Regarding > > the > > > > > current issue you are facing, here's my idea: If you could get > access > > > to > > > > > the web UI, you can get the hash (vertex id) in the url by clicking > > and > > > > > zooming in on the operator you want to query. IIUC, this hash can > be > > > used > > > > > to query the state. Is this feasible? Additionally, I think we > could > > > add > > > > > user-defined UIDs on the web UI and related REST APIs. Thus users > > could > > > > > easily identify an operator by uid, or get the uid of an operator. > > > > > > > > > > Best, > > > > > Zakelly > > > > > > > > > > On Thu, Aug 8, 2024 at 11:03 PM Gabor Somogyi < > > > gabor.g.somo...@gmail.com > > > > > > > > > > wrote: > > > > > > > > > > > Hi Zakelly, > > > > > > > > > > > > Thanks for the feedback, let me elaborate on this. > > > > > > > > > > > > In short Databricks has created a much more user friendly > > solution[1] > > > > for > > > > > > state observability (based on Flink's state processor API) than > > what > > > we > > > > > > have now. > > > > > > > > > > > > Up until now our state processor API was good enough but now > we're > > > > > lagging > > > > > > behind. We see users (just like Spark) where the first class > > citizen > > > is > > > > > the > > > > > > state itself and they're > > > > > > pointing to the new Spark solution. Since the state became first > > > class > > > > > > citizen there is a natural need to use it for business logic > > > > validation, > > > > > > debugging, explanatory browsing, etc... > > > > > > > > > > > > The main message here is that there are cases where users are not > > > able > > > > to > > > > > > identify operators because hash is a one way conversion. > > > > > > I'm open to any suggestion but somehow the initial operator human > > > > > readable > > > > > > identifier must be available. Let me come up with examples where > > > > > > users are completely blind. > > > > > > > > > > > > > Are you saying the user can set the operator uid but then > doesn't > > > > know > > > > > > what they set when debugging? > > > > > > > > > > > > There are cases where the user is setting the UID in the job, > such > > > case > > > > > > it's not user friendly to parse git repos but doable. > > > > > > But there are cases where the user has limited or no control > > related > > > > > UIDs: > > > > > > * SQL jobs are generating operators with meaningful names, but I > > > think > > > > > it's > > > > > > not realistic to enforce users to understand all the internals of > > > Flink > > > > > SQL > > > > > > implementation (which operator named where and how). > > > > > > * Iceberg is using the given UID as prefix and generating more > > > > operators > > > > > > with it > > > > > > * Weak justification but exists: Since operator name and UID are > > both > > > > > > optional some of the users are setting name only. Such case Flink > > > > > generates > > > > > > a random hash, where only name can give some pointers. > > > > > > > > > > > > Hope I've given better context. > > > > > > > > > > > > [1] > > > > > > > > > > > > > > > > > > > > > > > > > > > https://www.databricks.com/blog/announcing-state-reader-api-new-statestore-data-source > > > > > > > > > > > > BR, > > > > > > G > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Aug 8, 2024 at 12:06 PM Zakelly Lan < > zakelly....@gmail.com > > > > > > > > wrote: > > > > > > > > > > > > > Hi Gabor, > > > > > > > > > > > > > > Thanks for the proposal! However, I find it a little strange. > Are > > > you > > > > > > > saying the user can set the operator uid but then doesn't know > > what > > > > > they > > > > > > > set when debugging? Otherwise, is the > > > > > > `OperatorIdentifier.forUid("my-uid")` > > > > > > > feasible? I understand your point about potential cross-team > > work, > > > > but > > > > > > the > > > > > > > person may not be able to debug code that was not written by > > them. > > > > > Things > > > > > > > get complex in this scenario. Could you provide more details > > about > > > > the > > > > > > > issue you are facing? > > > > > > > > > > > > > > Regarding the checkpoint, it is not designed to be > self-contained > > > or > > > > > > > human-readable. I suggest not introducing such columns for > > > debugging > > > > > > > purposes. > > > > > > > > > > > > > > > > > > > > > Best, > > > > > > > Zakelly > > > > > > > > > > > > > > On Wed, Aug 7, 2024 at 10:07 PM Gabor Somogyi < > > > > > gabor.g.somo...@gmail.com > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > Hi Devs, > > > > > > > > > > > > > > > > I would like to start a discussion on FLIP-474: Store > operator > > > name > > > > > and > > > > > > > UID > > > > > > > > in state metadata[1]. > > > > > > > > > > > > > > > > In short users are interested in what kind of operators are > > > inside > > > > a > > > > > > > > checkpoint data which can be enhanced from user experience > > > > > perspective. > > > > > > > The > > > > > > > > details can be found in FLIP-474[1]. > > > > > > > > > > > > > > > > Please share your thoughts on this. > > > > > > > > > > > > > > > > [1] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-474%3A+Store+operator+name+and+UID+in+state+metadata > > > > > > > > > > > > > > > > BR, > > > > > > > > G > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
