Re: Manage multiple jobs in Flink

2020-11-03 Thread Robert Metzger
Hi Alexandru, 1. You can either create a Flink cluster per job (preferred), or use one big cluster to run all your jobs. This depends a bit on the resource manager you are using, and the workloads you are planning to process. If you are using Kubernetes, it makes sense to deploy each job separatel

Manage multiple jobs in Flink

2020-11-02 Thread Alexandru Vasiu
Hi, I have some questions: 1. How can you manage multiple jars (jobs) easily using Flink? 2. All jobs should run on the same task manager or do we need to use one for each job? 3. Can we store the jars in some persistent storage (such as S3) and start a job for each jar from that storage? 4. Also