Hi, We have a situation where we need to start a flink batch job on a yarn cluster the moment an event arrives over a queue. These events occur at a very low rate (like once or twice a week).
The idea we have is to run an application that listens to the queue and executes the batch when it receives a message. We found that if we start this using 'flink run -m yarn-cluster ..." the moment we run this the jobmanager in yarn is started and the resources for these batches is claimed immediately. What is the recommended way to only claim these resources when we actually have a job to run? Can we 'manually' start and stop the jobmanager in yarn in some way from our java code? -- Best regards / Met vriendelijke groeten, Niels Basjes
