Hi! The concrete storage location (HDFS / S3) should be independent of the imeplementation. The design should work for both.
Stephan On Mon, Mar 6, 2017 at 5:38 PM, Vishnu Viswanath < vishnu.viswanat...@gmail.com> wrote: > Thanks everyone for the reply. > > @Xiaogang design document looks very good, looking forward to check the > implementation and to try it out. I see that hdfs:// is mentioned as the > backend storage in the documentation, what about s3://. Is it something > that needed to be considered in the design, or is it handled by RocksDB. > > Thanks, > Vishnu > > On Sat, Mar 4, 2017 at 2:58 AM, SHI Xiaogang <shixiaoga...@gmail.com> > wrote: > > > Hi Vinshnu, > > > > We have obtained an initial design of incremental checkpointing [1] and > > will start working on incremental checkpointing the next week. You can > > watch the issue FLINK-5053 [2] to get timely notification of the > updates. > > All suggestions are welcome. > > > > [1] > > https://docs.google.com/document/d/1VvvPp09gGdVb9D2wHx6NX99yQK0jS > > UMWHQPrQ_mn520/edit#heading=h.2gshh42txc4z > > [2] https://issues.apache.org/jira/browse/FLINK-5053 > > > > Regards, > > Xiaogang > > > > > > > > 2017-03-04 15:43 GMT+08:00 Shaoxuan Wang <wshaox...@gmail.com>: > > > > > Vinshnu, > > > You can find the latest design discussion for incremental checkpoint in > > > http://apache-flink-mailing-list-archive.1008284.n3. > > > nabble.com/DISCUSS-Support-Incremental-Checkpointing-in- > > Flink-td15931.html > > > @Stefan Richter > > > <https://issues.apache.org/jira/secure/ViewProfile.jspa?name=srichter> > > > and @Xiaogang > > > Shi > > > <https://issues.apache.org/jira/secure/ViewProfile.jspa? > > name=xiaogang.shi> > > > might > > > be the right person to provide you its on-going status. > > > > > > cheers, > > > Shaoxuan > > > > > > On Sat, Mar 4, 2017 at 5:14 AM, Vishnu Viswanath < > > > vishnu.viswanat...@gmail.com> wrote: > > > > > > > Hi, > > > > > > > > Can someone point me to the branch where the ongoing work for > > incremental > > > > checkpoint is going on, I would like to try it out even if the work > is > > > not > > > > complete. > > > > > > > > I have a use case where the state size increase about ~1gb every 5 > > > minutes. > > > > > > > > Thanks, > > > > Vishnu > > > > > > > > > >
