Re: Use State query to dump state into datalake

2021-05-03 Thread David Anderson
I think you'd be better off using the State Processor API [1] instead. The State Processor API has cleaner semantics -- as you'll be seeing a self-consistent snapshot of all the state -- and it's also much more performant. Note also that the Queryable State API is "approaching end of life" [2]. Th

Use State query to dump state into datalake

2021-05-02 Thread Lian Jiang
Hi, I am interested in dumping Flink state from Rockdb to datalake using state query https://ci.apache.org/projects/flink/flink-docs-master/docs/dev/datastream/fault-tolerance/queryable_state/. My map state could have 200 million key-values pairs and the total size could be 150G bytes. My batch jo