Hi, All At present flink have three state backend: memory, file system, rocksdb. MemoryStateBackend will tansform the snapshot to jobManager, 5MB limited default. Even setting it bigger, that not suitable for very big state storage. HDFS can meet the reliability guarantee, but It's slow. File System and RocksDB are fast, but they are have no reliability guarantee. Three state backend all have no reliability guarantee.
Can we have a Hbase state backend, providing reliability guarantee of state snapshot? For user, only new a HbaseStateBackend object, provide hbase parameter and optimization configure. Maybe Hbase or other distributed key-value storage is heavyweight storage, we only use hbase client to read/write asynchronously. -Jinkui Shi
