RE: flink memory management / temp-io dir question

2018-10-11 Thread anand.gopinath
Its ok – I see the relevant docs now… "The RocksDBStateBackend holds in-flight data in a RocksDB database that is (per default) stored in the TaskManager data directories." Thanks for your help Anand From: Gopinath, Anand Sent: 11 October 2018 18:40 To: 'Till Rohrmann'; Kostas Kloudas Cc: user;

RE: flink memory management / temp-io dir question

2018-10-11 Thread anand.gopinath
Hi Till, Thanks for the reply. I don’t use batch, so I assume what I am seeing is streaming related. I thought rocksdb writes to a different dir though ( as defined by checkpoint.data.uri)? Regards, Anand From: Till Rohrmann [mailto:trohrm...@apache.org] Sent: 08 October 2018 13:39 To: Kostas

Are savepoints / checkpoints co-ordinated?

2018-10-11 Thread anand.gopinath
Hi, I had a couple questions about savepoints / checkpoints When I issue "Cancel Job with Savepoint", how is that instruction co-ordinated with check points? Am I certain the savepoint will be the last operation (i.e. no more check points)? I have a kafka src>operation>kafka sink task in flink

flink memory management / temp-io dir question

2018-10-05 Thread anand.gopinath
Hi , I had a question with respect flink memory management / overspill to /tmp. In the docs (https://ci.apache.org/projects/flink/flink-docs-release-1.6/ops/config.html#configuring-temporary-io-directories) it says: Although Flink aims to process as much data in main memory as possible, it is n