Hey, Yes, I agree that it is probably best to remove this backend from Flink and make it an external library instead.
I will prepare a JIRA and PR. Gyula Aljoscha Krettek <aljos...@apache.org> ezt írta (időpont: 2016. márc. 16., Sze, 10:35): > Hi, > if you yourself (Gyula) don’t want to maintain it anymore in the Flink > codebase I would vote to move it to an external repository. If you are not > using it anymore I’m afraid no one will really work on it. > > On more thing. When using the DB state backend savepoints don’t work. > Cleanup/compaction mess with the stored state and a checkpoint/savepoint of > the DB state backend is therefore not a self-contained unit. > > Cheers, > Aljoscha > > On 14 Mar 2016, at 12:30, Ufuk Celebi <u...@apache.org> wrote: > > > > On Mon, Mar 14, 2016 at 11:20 AM, Gyula Fóra <gyf...@apache.org> wrote: > >> We developed (and contributed) the DB state backend to be able to handle > >> large state sizes, but have moved to RocksDB for much better read/write > >> performance. I have a pending PR with some improvements to the MySQL > >> adapter but the question is whether we should keep this as part of Flink > >> contrib or move it to an external library. > >> > >> I am personally undecided whether there are strong use cases that would > >> favour this backend over RocksDB. > > > > Hey Gyula, > > > > thanks for starting this discussion and the great summary of Pros and > > Cons. I fully agree that it boils down to whether there are strong use > > cases that would favour the DB backend over RocksDB. > > > > Personally, I am in favour of moving the DB state backend out of core > Flink: > > > > - I think that the DB state backend was a very valuable contribution, > > but as you've already outlined, RocksDB seems to work better for large > > out of core state, which was the prime motivation for the DB state > > backend. > > > > - Furthermore, my experience from chatting with you while you were > > evaluating the DB backend was that there are many parameters to tune > > for this to work realiably (at least with MySQL). I'm wondering how > > many users will have the same motivation to actually look into all of > > these parameters. In my experience, RocksDB provides a better out of > > the box experience, which is in line with Flink's over all philosophy. > > > > It would be possible to just keep the code around in the contrib > > module, but if we don't have the resources to maintain it (including > > more documentation and tests), I don't see the point. > > > > Since you are the main person driving this, the question is how much > > time you can/want to invest into this. If you feel like that you > > personally would really like to keep it in Flink and develop it > > further here, I would certainly be in favour of keeping it. But at the > > moment, I don't see this being the case. > > > > – Ufuk > >
