I’m currently trying to programmatically create a Flink cluster on a given YARN cluster. I’m using the FlinkYarnClientBase class to do this currently with some limitations (Flink version 1.0.3).
I’m wanting to pass in my own YARN configuration so that I can deploy Flink on different YARN clusters. Currently these appear to be picked up from an environment variable YARN_CONF or if I add the yarn configuration files into the classpath it works for that one. Is there a way I can dynamically build the YARN configuration and Flink configuration without utilizing the file system? It would be nice if we had the option to utilize the same Flink client code and have the CLI be a wrapper that adds all the environment variable stuff, and then when we need to do create these programmatically we can do that too. I’ve started looking at the new code in Flink master and it seems a little easier than in 1.0.3 but seems that the YarnConfiguration is private in AbstractYarnClusterDescriptor making it difficult to set the configuration programmatically. I’m not sure if the new client code still throws an exception if the Flink config doesn’t exist, even though I could build out the Flink config programmatically. Is there a recommended approach to allow this? Benjamin
