Re: Memory Leak - Flink / RocksDB ?

2017-07-27 Thread Shashwat Rastogi
Hi Kien,Sorry it took me sometime to fetch the logs. I am attaching logs of the machine which died due to lack of free memory. Jun 26 13:00:01 staging-east-dataplatform-02-c01 systemd: Created slice user-0.slice. Jun 26 13:00:01 staging-east-dataplatform-02-c01 systemd: Starting user-0.slice. Jun

Re: Memory Leak - Flink / RocksDB ?

2017-07-26 Thread Shashwat Rastogi
Hi Vinay, @Vinay : I am setting RocksDBStateBackend from the code, not from flink-conf.yaml. I am currently trying out the configurations that you have shared. I’ll let you know how they perform. Thank you so much for your help. However, were you able to figure out what exactly is going wrong

Memory Leak - Flink / RocksDB ?

2017-07-25 Thread Shashwat Rastogi
Hi, We have several Flink jobs, all of which reads data from Kafka do some aggregations (over sliding windows of (1d, 1h)) and writes data to Cassandra. Something like : ``` DataStream lines = env.addSource(new FlinkKafkaConsumer010( … )); DataStream events = lines.map(line -> parse(line)); Dat