General guidance

2021-03-25 Thread Almeida, Julius
Hi Team, My streaming pipeline is based on beam & running using flink runner with rocksdb as state backend. Over time I am seeing memory spike & after giving a look at heap dump, I am seeing records in ‘__StatefulParDoGcTimerId’ which seems to be never cleaned. Found this jira https://issues

State size increasing exponentially in Flink v1.9

2021-03-24 Thread Almeida, Julius
Hey, Hope you all are doing well! I am using flink v1.9 with RocksDBStateBackend, but over time the state size is increasing exponentially. I am using MapState in my project & seeing memory spike, after looking at heap dump I see duplicates in it. I also have logic added to remove expired even