IIRC, all the above support data locality from back in the MR days. Not sure how much data you're planning to checkpoint though -- is locality really that important for transient processor state?
On Thu, Jun 9, 2016 at 11:06 AM, CPC <acha...@gmail.com> wrote: > Cassandra backend would be interesting especially if flink could benefit > from cassandra data locality. Cassandra/spark integration is using this for > information to schedule spark tasks. > > On 9 June 2016 at 19:55, Nick Dimiduk <ndimi...@gmail.com> wrote: > > > You might also consider support for a Bigtable > > backend: HBase/Accumulo/Cassandra. The data model should be similar > > (identical?) to RocksDB and you get HA, recoverability, and support for > > really large state "for free". > > > > On Thursday, June 9, 2016, Chen Qin <qinnc...@gmail.com> wrote: > > > > > Hi there, > > > > > > What is progress on incremental checkpointing? Does flink dev has plan > to > > > work on this or JIRA to track this? super interested to know. > > > > > > I also research and consider use rocksdbstatebackend without running > HDFS > > > cluster nor talk to S3. Some primitive idea is to use ZK to store / > > notify > > > state propagation progress and propagate via implement chain > replication > > on > > > top of YARN provisioned storage node. > > > > > > Thanks, > > > Chen > > > > > >
