Hi Ken and Rong, That's a good point. The FSStateBackend holds all state in the JVM heap as well. The name is also not well chosen, IMO.
Thanks for creating the JIRA ticket! Fabian 2018-05-05 0:52 GMT+02:00 Ken Krugler <kkrugler_li...@transpac.com>: > > > On May 3, 2018, at 9:06 PM, Rong Rong <walter...@gmail.com> wrote: > > > > Agree it is definitely not intuitive trying to figure out what to do > based > > on this message. > > I think the message should be changed to "please consider increasing > > maximum permitted memory size, increase task manager parallelism, or > using > > a non-memory-based state backend". > > > > Could you please open a JIRA ticket in: > > https://issues.apache.org/jira/projects/FLINK/issues/ > > Sure, see > > https://issues.apache.org/jira/browse/FLINK-9300 < > https://issues.apache.org/jira/browse/FLINK-9300> > > — Ken > > > On Thu, May 3, 2018 at 7:56 PM, Ken Krugler <kkrugler_li...@transpac.com > > > > wrote: > > > >> Currently in the MemCheckpointStreamFactory.checkSize() method, it can > >> throw an IOException with: > >> > >> throw new IOException( > >> "Size of the state is larger than > >> the maximum permitted memory-backed state. Size=" > >> + size + " , > >> maxSize=" + maxSize > >> + " . Consider > >> using a different state backend, like the File System State backend."); > >> > >> But this will happen even if you’re using the File System State backend, > >> right? > >> > >> This came up here: https://stackoverflow.com/questions/50149005/ > >> ioexception-size-of-the-state-is-larger-than-the-maximum- > >> permitted-memory-backe <https://stackoverflow.com/questions/50149005/ > >> ioexception-size-of-the-state-is-larger-than-the-maximum- > >> permitted-memory-backe> > >> > >> So shouldn’t it suggest using the RocksDB state backend instead? > >> > >> — Ken > > -------------------------------------------- > http://about.me/kkrugler > +1 530-210-6378 > >
