Hello everyone! I would like to start a discussion regarding the future of the Database state backend for the streaming API. The main question is whether we want to keep this as a flink-contrib moduel and continue development as part of Flink or should we move it to an outside library.
Just as a quick refresher, the DB state backend is designed to store streaming state in JDBC supporting databases (the connector in the flink repo supports MySQL). There is an adapter interface that can be implemented to use different databases other than MySQL. Advantages: - Can store huge states (tested with more than a TB of state with MySQL) - Fast, incremental checkpoints by default - In-memory cache for hot keys - Supports rescaling easily (restarting the job with different parallelism would still work as expected) - Supports database sharding with custom data partitioner - Exposes an easily accessible state view to the outside (will contain uncheckpointed state) Disadvantages: - Lookup/Update might be much slower than local state backend (RocksDB for instance) - Incremental checkpoints are not very well supported with the current runtime so state cleanup/compaction does not happen automatically - Restore speed depends on database indexing as it involves range deletes based on timestamps (this might get slow on very large states) - The caching layer is not memory aware, this should be moved to flink managed memory We developed (and contributed) the DB state backend to be able to handle large state sizes, but have moved to RocksDB for much better read/write performance. I have a pending PR with some improvements to the MySQL adapter but the question is whether we should keep this as part of Flink contrib or move it to an external library. I am personally undecided whether there are strong use cases that would favour this backend over RocksDB. Any opinions? Cheers, Gyula
