Thanks everyone for the reply. @Xiaogang design document looks very good, looking forward to check the implementation and to try it out. I see that hdfs:// is mentioned as the backend storage in the documentation, what about s3://. Is it something that needed to be considered in the design, or is it handled by RocksDB.
Thanks, Vishnu On Sat, Mar 4, 2017 at 2:58 AM, SHI Xiaogang <shixiaoga...@gmail.com> wrote: > Hi Vinshnu, > > We have obtained an initial design of incremental checkpointing [1] and > will start working on incremental checkpointing the next week. You can > watch the issue FLINK-5053 [2] to get timely notification of the updates. > All suggestions are welcome. > > [1] > https://docs.google.com/document/d/1VvvPp09gGdVb9D2wHx6NX99yQK0jS > UMWHQPrQ_mn520/edit#heading=h.2gshh42txc4z > [2] https://issues.apache.org/jira/browse/FLINK-5053 > > Regards, > Xiaogang > > > > 2017-03-04 15:43 GMT+08:00 Shaoxuan Wang <wshaox...@gmail.com>: > > > Vinshnu, > > You can find the latest design discussion for incremental checkpoint in > > http://apache-flink-mailing-list-archive.1008284.n3. > > nabble.com/DISCUSS-Support-Incremental-Checkpointing-in- > Flink-td15931.html > > @Stefan Richter > > <https://issues.apache.org/jira/secure/ViewProfile.jspa?name=srichter> > > and @Xiaogang > > Shi > > <https://issues.apache.org/jira/secure/ViewProfile.jspa? > name=xiaogang.shi> > > might > > be the right person to provide you its on-going status. > > > > cheers, > > Shaoxuan > > > > On Sat, Mar 4, 2017 at 5:14 AM, Vishnu Viswanath < > > vishnu.viswanat...@gmail.com> wrote: > > > > > Hi, > > > > > > Can someone point me to the branch where the ongoing work for > incremental > > > checkpoint is going on, I would like to try it out even if the work is > > not > > > complete. > > > > > > I have a use case where the state size increase about ~1gb every 5 > > minutes. > > > > > > Thanks, > > > Vishnu > > > > > >
