It seems like another option here would be to occasionally use the state processor API to purge a savepoint of all unnecessary state.
On Fri, Oct 30, 2020 at 6:57 PM Steven Wu <stevenz...@gmail.com> wrote: > not a solution, but a potential workaround. Maybe rename the operator uid > so that you can continue to leverage allowNonRestoredState? > > On Thu, Oct 29, 2020 at 7:58 AM Peter Westermann < > no.westerm...@genesys.com> wrote: > >> Does that actually allow removing a state completely (vs. just modifying >> the values stored in state)? >> >> >> >> Ideally, we would want to just interact with state via *KeyedStateStore*. >> Maybe it would be possible to add a couple methods there, e.g. like this: >> >> // List all pre-existing states >> >> <S extends State, T> List<StateDescriptor<S, T>> listStates(); >> >> // Completely remove a state >> >> <S extends State, T> void dropState(StateDescriptor<S, T> >> stateDescriptor); >> >> >> >> >> >> Thanks, >> >> Peter >> >> >> >> >> >> >> >> *From: *Congxian Qiu <qcx978132...@gmail.com> >> *Date: *Thursday, October 29, 2020 at 10:38 AM >> *To: *Robert Metzger <rmetz...@apache.org> >> *Cc: *Peter Westermann <no.westerm...@genesys.com>, " >> user@flink.apache.org" <user@flink.apache.org> >> *Subject: *Re: Feature request: Removing state from operators >> >> >> >> Hi Peter >> >> Can applyToAllKeys[1] in KeyedStateBackend help you here? but >> currently, this is not exposed to users now. >> >> >> >> [1] >> https://github.com/apache/flink/blob/fada6fb6ac9fd7f6510f1f2d77b6baa06563e222/flink-runtime/src/main/java/org/apache/flink/runtime/state/KeyedStateBackend.java#L65 >> >> >> Best, >> >> Congxian >> >> >> >> >> >> Robert Metzger <rmetz...@apache.org> 于2020年10月27日周二 下午5:51写道： >> >> Hi Peter, >> >> >> >> I'm adding two committers to this thread who can help answering your >> question. >> >> >> >> On Mon, Oct 26, 2020 at 3:22 PM Peter Westermann < >> no.westerm...@genesys.com> wrote: >> >> We use the feature for removing stateful operators via the >> *allowNonRestoredState* relatively often and it works great. However, >> there doesn’t seem to be anything like that for removing state from an >> existing operator (that we want to keep). >> >> Say my operator defines a *MapState* and a *ValueState*. Later on, the >> *ValueState* becomes obsolete. In this case, we can remove the actual >> data for each key by clearing it out but the state itself is still >> referenced in savepoints even if it’s not referenced in code anymore – that >> e.g. means one cannot remove any class that was previously used in state. >> >> Would it be possible to add support for completely removing state from an >> operator if it’s no longer referenced in code and *allowNonRestoredState* >> is set? (Or to add an explicit “drop this state option” in KeyedStateStore >> and OperatorStateStore?) >> >> >> >> Thanks, >> >> Peter >> >> >> >>
