Hi, also +1. As vivo said, SavePoint is not compatible. I have heard from a lot of users just said "my previous program does not work any more!". If these utilities provide such migration functions, it would be perfect!
Best, tison. vino yang <yanghua1...@gmail.com> 于2018年8月18日周六 上午11:04写道： > Hi, > > +1, from my side. Considering that Savepoint is not compatible with every > version of Flink. > It would be very useful if it could do some conversion between different > versions of Flink Savepoint. > > Thanks, vino. > > bupt_ljy <bupt_...@163.com> 于2018年8月18日周六 上午2:55写道： > > > Hi, > > +1, I think it will be a very great tool for Flink, especially the > > creating new state part. On production, we’re really worried about the > > availability of the savepoints, because the generating logic is inside > > Flink and we don’t have a good way to validate it. But with this tool, we > > can construct a new state for our programs very soon even if the > savepoints > > data is broken. > > It’s great, thanks! > > > > > > Original Message > > Sender:Jamie grierjgr...@lyft.com > > Recipient:dev...@flink.apache.org > > Date:Saturday, Aug 18, 2018 02:32 > > Subject:Re: [Proposal] Utilities for reading, transforming and > > creatingStreaming savepoints > > > > > > This is great, Gyula! A colleague here at Lyft has also done some work > > around bootstrapping DataStream programs and we've also talked a bit > about > > doing this by running DataSet programs. On Fri, Aug 17, 2018 at 3:28 AM, > > Gyula Fóra gyula.f...@gmail.com wrote: Hi All! I want to share with > > you a little project we have been working on at King (with some help > from > > some dataArtisans folks). I think this would be a valuable addition to > > Flink and solve a bunch of outstanding production use-cases and > headaches > > around state bootstrapping and state analytics. We have built a quick > and > > dirty POC implementation on top of Flink 1.6, please check the README > for > > some nice examples to get a quick idea: https://github.com/king/bravo > > *Short story* Bravo is a convenient state reader and writer library > > leveraging the Flink’s batch processing capabilities. It supports > > processing and writing Flink streaming savepoints. At the moment it only > > supports processing RocksDB savepoints but this can be extended in the > > future for other state backends and checkpoint types. Our goal is to > > cover a few basic features: - Converting keyed states to Flink DataSets > > for processing and analytics - Reading/Writing non-keyed operators > states > > - Bootstrap keyed states from Flink DataSets and create new valid > > savepoints - Transform existing savepoints by replacing/changing some > > states Some example use-cases: - Point-in-time state analytics > across > > all operators and keys - Bootstrap state of a streaming job from > external > > resources such as reading from database/filesystem - Validate and > > potentially repair corrupted state of a streaming job - Change max > > parallelism of a job Our main goal is to start working together with > > other Flink production users and make this something useful that can be > > part of Flink. So if you have use-cases please talk to us :) I have > also > > started a google doc which contains a little bit more info than the > readme > > and could be a starting place for discussions: > > https://docs.google.com/document/d/103k6wPX20kMu5H3SOOXSg5PZIaYpw > > dhqBMr-ppkFL5E/edit?usp=sharing I know there are a bunch of rough edges > > and bugs (and no tests) but our motto is: If you are not embarrassed, > you > > released too late :) Please let me know what you think! Cheers, > Gyula >
