Re: State backend considerations

Hi Nick, Both questions are hard to answer given that it depends on your hardware, access patterns (read/update), record size/structure, parallelism, and probably a ton of other parameters. The usual approach is to simply evaluate it in your setting. Since it's a matter of configuration, you can

Re: State backend configuration

Hi James, the configuration options were consolidates a bit for Flink 1.5. What was called state.backend.fs.checkpointdir before was the same as state.checkpoints.dir, we now made this the same for the two backends. Also, the file-backend does not need local storage on the worker nodes, that's

Re: state backend configuration

Hi, This is not possible on versions of Flink prior to 1.5 but starting with that (as of yet unreleased) versions you'll be able to configure this via state.backend.incremental: true. Configurations that you do in code will take precedence. Best, Aljoscha > On 14. Mar 2018, at 02:33, Jayant A

Re: State backend questions

According to this thread it is not yet possible to switch to/from RocksDBStatebackend, so I would suggest to with RocksDB from the start. For tuning RocksDB, see here

Re: State Backend

Hi, if the question is, if there are certain requirements for the filesystem that you use with the state backends, then I think there might be a small misconception. Currently, all state backends in Flink operator local to the task, i.e. either in memory (e.g. FsStateBackend) or also on the loc