On Mon, Mar 14, 2016 at 11:20 AM, Gyula Fóra <gyf...@apache.org> wrote: > We developed (and contributed) the DB state backend to be able to handle > large state sizes, but have moved to RocksDB for much better read/write > performance. I have a pending PR with some improvements to the MySQL > adapter but the question is whether we should keep this as part of Flink > contrib or move it to an external library. > > I am personally undecided whether there are strong use cases that would > favour this backend over RocksDB.
Hey Gyula, thanks for starting this discussion and the great summary of Pros and Cons. I fully agree that it boils down to whether there are strong use cases that would favour the DB backend over RocksDB. Personally, I am in favour of moving the DB state backend out of core Flink: - I think that the DB state backend was a very valuable contribution, but as you've already outlined, RocksDB seems to work better for large out of core state, which was the prime motivation for the DB state backend. - Furthermore, my experience from chatting with you while you were evaluating the DB backend was that there are many parameters to tune for this to work realiably (at least with MySQL). I'm wondering how many users will have the same motivation to actually look into all of these parameters. In my experience, RocksDB provides a better out of the box experience, which is in line with Flink's over all philosophy. It would be possible to just keep the code around in the contrib module, but if we don't have the resources to maintain it (including more documentation and tests), I don't see the point. Since you are the main person driving this, the question is how much time you can/want to invest into this. If you feel like that you personally would really like to keep it in Flink and develop it further here, I would certainly be in favour of keeping it. But at the moment, I don't see this being the case. – Ufuk
