Re: Stop job with savepoint during graceful shutdown on a k8s cluster

2020-03-16 Thread Vijay Bhaskar
For point (1) above: Its up to user to have proper sink and source to choose to have exactly once semantics as per the documentation: https://ci.apache.org/projects/flink/flink-docs-stable/dev/connectors/guarantees.html If we choose the supported source and sink combinations duplicates will be avoi

Re: Stop job with savepoint during graceful shutdown on a k8s cluster

2020-03-14 Thread shravan
Our understanding is to stop job with savepoint, all the task manager will persist their state during savepoint. If a Task Manager receives a shutdown signal while savepoint is being taken, does it complete the savepoint before shutdown ? [Ans ] Why task manager is shutdown suddenly? Are you saying

Re: Stop job with savepoint during graceful shutdown on a k8s cluster

2020-03-13 Thread Vijay Bhaskar
Please find answers inline Our understanding is to stop job with savepoint, all the task manager will persist their state during savepoint. If a Task Manager receives a shutdown signal while savepoint is being taken, does it complete the savepoint before shutdown ? [Ans ] Why task manager is shutd

Stop job with savepoint during graceful shutdown on a k8s cluster

2020-03-13 Thread shravan
Job Manager , Task Manager are run as separate pods within K8S cluster in our setup. As job cluster is not used, job jars are not part of Job Manager docker image. The job is submitted from a different Flink client pod. Flink is configured with RocksDB state backend. The docker images are created