Hi Yu,
Very cool! I might be out of dated of what’s new in Flink already…
Just wonder If there are efforts to support seconds level barrier alignment?
Chen
> On Dec 27, 2018, at 23:26, Yu Li wrote:
>
> FWIW, one major advantage of adopting HBase as Flink statebackend is to
> support direct re
FWIW, one major advantage of adopting HBase as Flink statebackend is to
support direct read/write on DFS, so as to disaggregate storage and compute
(DisAgg). DisAgg has several benefits, such as supporting elastic
computing in cloud, much better (order of magnitude) recovery speed when
rescaling u
Hi Naveen,
AFAIK, there are two level of storage in typical statebackend
(local/remote). I think it kinda similar to what PC main memory and disk
analogy.
Take RocksDB Statebackend as example, window state (typical very large
ListState) persisted in partitioned local rocksdb files, adding element
Did try to use rocksdb[1] as state backend?
1.
https://ci.apache.org/projects/flink/flink-docs-stable/ops/state/state_backends.html#the-rocksdbstatebackend
On Thu, 27 Dec 2018, 18:17 Naveen Kumar Hi,
>
> I am exploring if we can plugin hbase as state backend in Flink. We have
> need for stream
Hi!
While certainly possible I think it’s a bad idea in general.
I think state size itself shouldn’t be a problem with the RocksDb backend
as you can always increase parallelism to shard more while keeping the
insanely good performance compared to a remote kv store. We and other users
have succes
Hi,
I am exploring if we can plugin hbase as state backend in Flink. We have
need for streaming jobs with large window states, high throughput and
reliability.
I wanted to know if implementing Flink backend in Hbase or other
distributed KV store is possible. Any documentation or pointers will be
