Re: State Recovery when job fails and auto-recovers

2018-10-17 Thread Hequn Cheng
Hi Sameer, In case of a failure, the job will restarts the operators and resets them to the latest successful checkpoint. So if you turn off checkpoints, all data will be lost. Generally speaking, snapshots are very light-weight and can be drawn frequently without much impact on performance. If it

State Recovery when job fails and auto-recovers

2018-10-17 Thread Sameer Wadkar
Hi, We have a job which is using ValueState. We have turned off checkpoints. The state is backed by rocksdb which is backed by S3. If the job fails for any exception (ex. Partitions not available or an occasional S3 404 error) and auto-recovers, is the entire state lost or does it continue f