Hi Yu, For the first question, I would say yes. I was talking about managed states, to be more specific, it's managed keyed states. And the reason why we need the framework to manage life cycle is that we need checkpoint to guarantee exact once semantic in our customized keyed state backend.
For the second question, I am quite agree with your proposal. Finally, I would be glad to provide documentation if needed. Best, Shimin Yun Tang <myas...@live.com> 于2019年9月9日周一 上午2:46写道： > Hi all > > First of all, I agreed with Yu that we should support to make state type > pluginable. > > If we take a look at current Flink implementation. Users could implement > their pluginable state backend to satisfy their own meets now. However, > even users could define their own state descriptor, they cannot store the > customized state within their state backend. The root cause behind this is > that current StateBackendFactory could accept user defined state backend > factory while StateFactory (within HeapKeyedStateBackend [1] and > RocksDBKeyedStateBackend [2] ) cannot. > > If we agreed that we should leave the right of implementing customized > state backend to users, it's naturally to agree that we should also leave > the right of implementing customized states to users. > > [1] > https://github.com/apache/flink/blob/576228651382db040aaa006cf9142f6568930cb1/flink-runtime/src/main/java/org/apache/flink/runtime/state/heap/HeapKeyedStateBackend.java#L79 > [2] > https://github.com/apache/flink/blob/576228651382db040aaa006cf9142f6568930cb1/flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/RocksDBKeyedStateBackend.java#L114 > > > Best > Yun Tang > > > ________________________________ > From: Yu Li <car...@gmail.com> > Sent: Monday, September 9, 2019 2:24 > To: dev <dev@flink.apache.org> > Subject: Re: [DISCUSS] Support customize state in customized > KeyedStateBackend > > Hi Shimin, > > Thanks for bring this discussion up. > > First of all, I'd like to confirm/clarify that this discussion is mainly > about managed state with customized state descriptor rather than raw state, > right? Asking because raw state was the very first thing came to my mind > when seeing the title. > > And this is actually the first topic/question we need to discuss, that > whether we should support user-defined state descriptor and still ask > framework to manage the state life cycle. Personally I'm +1 on this since > the "official" state (data-structure) types (currently mainly value, list > and map) may not be optimized for customer case, but we'd better ask > others' opinion. > > Secondly, if the result of the first question is "Yes", then it's truly a > problem that "Although we can implement customized StateDescriptors for > different kind of data structures, we do not really have access to such > customized state in RichFunctions", and how to resolve it is the second > topic/question to discuss. > > I've noticed your proposal of exposing "getParitionedState" method out in > "RuntimeContext" and "KeyedStateStore" in JIRA (FLINK-14003), but IMO > adding a specific interface like below is better than exposing the internal > one: > <S extends State, V> State getCustomizedState(StateDescriptor<S, V> > stateProperties); > > Finally, I think this is a user-facing and definitely worthwhile > discussion, and requires a FLIP to document the conclusion and > design/implementation (if any) down. What's your opinion? > > Thanks. > > Best Regards, > Yu > > > On Fri, 6 Sep 2019 at 13:27, shimin yang <ysmcl...@gmail.com> wrote: > > > Hi every, > > > > I would like to start a discussion on supporting customize state > > in customized KeyedStateBackend. > > > > In Flink, users can customize KeyedStateBackend to support different type > > of data store. Although we can implement customized StateDescriptors for > > different kind of data structrues, we do not really have access to such > > customized state in RichFunctions. > > > > I propose to add a getOtherState method in RuntimeContext and > > DefaultKeyedStateStore which directly takes StateDescriptor as parameter > to > > allow user to get customized state. > > > > What do you think? > > > > Thanks. > > > > Best, > > Shimin > > >
