@Yun yes, this is really about making CheckpointStorage an orthogonal concept. I think we can remain pragmatic and keep state-backend specific configurations (async, incremental, etc) in the state backend themselves. I view these as more advanced configurations and by the time someone is changing the defaults they likely understand what is going on. My goal is to help on-board users and so long as each state backend has a no-arg default constructor that works for many users I think we've achieved that goal.
Regarding the checkpoint coordinator, that makes sense but I will consider out of the scope of this FLIP. I want to focus on simplifying APIs. @Aljoscha Krettek <aljos...@apache.org> My feeling is that state backends and checkpointing are going to be integral to Flink for many years, regardless or other enhancements so this change is still valuable. Since this is a FLIP about improving the user api I'm happy to bikeshed the names a little more than normal. HashMap makes sense, my other thought was InMemory. Seth On Wed, Sep 9, 2020 at 8:04 AM Aljoscha Krettek <aljos...@apache.org> wrote: > I like it a lot! > > I think it makes sense to clean this up despite the planned new > fault-tolerance mechanisms. In the future, users will decide which > mechanism to use and I can imagine that a lot of them will keep using > the current mechanism for quite a while to come. But I'm happy to yield > to Stephan's opinion here, he knows more about the progress of that work. > > The one nitpick I have is about naming: will users understand > OnHeapStateBackend? I mean, do they know what on-heap/off-heap memory is > and the tradeoffs? An alternative could be HashMapStateBackend, because > that's essentially what it is. I wouldn't block anything on this, though. > > Aljoscha > > On 09.09.20 10:05, Konstantin Knauf wrote: > > Thanks for the initiative. Big +1. Would be interested to hear if the > > proposed interfaces still make sense in the face of the new > fault-tolerance > > work that is planned. Stephan/Piotr will know. > > > > On Tue, Sep 8, 2020 at 7:05 PM Seth Wiesman <sjwies...@gmail.com> wrote: > > > >> Hi Devs, > >> > >> I'd like to propose an update to how state backends and checkpoint > storage > >> are configured to help users better understand Flink. > >> > >> Apache Flink's durability story is a mystery to many users. One of the > most > >> common recurring questions from users comes from not understanding the > >> relationship between state, state backends, and snapshots. Some of this > >> confusion can be abated with learning material but the question is so > >> pervasive that we believe Flink’s user APIs should be better communicate > >> what different components are responsible for. > >> > >> > >> > >> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-142%3A+Disentangle+StateBackends+from+Checkpointing > >> > >> > >> I look forward to a healthy discussion. > >> > >> > >> Seth > >> > > > > > >
