Re: Reg Checkpoint size using RocksDb

2016-12-19 Thread Anirudh Mallem
lto:user@flink.apache.org>" Date: Monday, December 19, 2016 at 11:51 AM To: "user@flink.apache.org<mailto:user@flink.apache.org>" Subject: Re: Reg Checkpoint size using RocksDb Hi! If you use the default checkpoint mode, Flink will checkpoint the current RocksDB instance. It may be

Re: Reg Checkpoint size using RocksDb

2016-12-19 Thread Stephan Ewen
Hi! If you use the default checkpoint mode, Flink will checkpoint the current RocksDB instance. It may be that there simply has not been a compaction in RocksDB when checkpointing, so the checkpoint contains some "old data" as well. If you switch to the "fully async" mode, it should always only c

Reg Checkpoint size using RocksDb

2016-12-19 Thread Anirudh Mallem
Hi, I was experimenting with using RocksDb as the state backend for my job and to test its behavior I modified the socket word count program to store states. I also wrote a RichMapFunction which stores the states as a ValueState with default value as null. What the job does basically is, for eve