Typically, when one wants to run a Flink job on a Hadoop YARN installation, one creates a Yarn session (e.g. ./bin/yarn-session.sh -n 4 -qu test-yarn-queue) and runs intended Flink job(s) (e.g. ./bin/flink run -c MyFlinkApp -m job-manager-host:job-manager-port <overriding app config params> myapp.jar) on the Flink cluster whose job manager URL is returned by the previous command.
My questions are: - Does yarn-session.sh need conf/flink-conf.yaml to be available in Flink installation on every container in YARN? If this file is needed, how can one run different YARN sessions (with potentially very different configurations) on the same Hadoop YARN installation simultaneously? - Is it possible to start the YARN session programmatically? If yes, I believe I should look at classes like YarnClusterClient <https://ci.apache.org/projects/flink/flink-docs-stable/api/java/org/apache/flink/yarn/YarnClusterClient.html>. Is that right? Is there any other guidance on how to do this programmatically (e.g. I have a management UI that wants to start/stop YARN sessions and deploy Flink jobs to it)? Regards, Kedar
