Hi all, While drafting FLIP-74 I notice that a job level client(called JobClient) is always retrieved from a Flink application cluster level client(called ClusterClient), which is then always retrieved from a extern cluster(YARN, mesos, k8s, etc.) level client(called ClusterDescriptor).
A Flink job management platform possibly requires all levels of client mentioned above for customized Flink cluster deployment, Flink cluster management, job submission and job management. They can be sorted into interfaces of different level of client mentioned above. ClusterDescriptor: Flink cluster deployment ClusterClient(retrieved from ClusterDescriptor): Flink cluster management and job submission JobClient(retrieved from ClusterClient): job management Recently we have FLIP-73 and FLIP-74 working on client API side, FLIP-74 is aimed at the job level client called JobClient which take responsibility of job management. FLIP-73 is aimed at a dedicated job executor which take responsibility of job submission. However, for full functions to handle Flink clusters & jobs, Flink cluster management and Flink cluster deployment still require public interface.These interface is mainly used for downstream project developer instead of users who are only interested in Flink job. Further, we already used ClusterClient and ClusterDescriptor in CliFrontend, sql-client, scala-shell and so on, which can be regarded as downstream project hosted in Flink repo. Given this observation, there is no reason we don't expose API for Flink cluster management and Flink cluster deployment. So here comes two question: 1. Do we want to expose multiple level clients as described above? If no, why? If so, what does the plan look like? 2. Does Executor introduced in the ongoing FLIP-73 break multiple level clients layout described above? Since ClusterClient already implements functions for job submission, where is Executor in the layout above? Does it overlap with existing client concept? Best, tison.
