Hi Tison, I shared some thoughts on this topic in the FLIP-74 thread, as some of the questions are related to that.
Cheers, Kostas On Wed, Sep 25, 2019 at 12:30 PM Zili Chen <wander4...@gmail.com> wrote: > > Hi all, > > While drafting FLIP-74 I notice that a job level client(called JobClient) is > always retrieved from a > Flink application cluster level client(called ClusterClient), which is then > always retrieved from a > extern cluster(YARN, mesos, k8s, etc.) level client(called ClusterDescriptor). > > A Flink job management platform possibly requires all levels of client > mentioned above for > customized Flink cluster deployment, Flink cluster management, job submission > and job > management. They can be sorted into interfaces of different level of client > mentioned above. > > ClusterDescriptor: Flink cluster deployment > ClusterClient(retrieved from ClusterDescriptor): Flink cluster management and > job submission > JobClient(retrieved from ClusterClient): job management > > Recently we have FLIP-73 and FLIP-74 working on client API side, FLIP-74 is > aimed at the job > level client called JobClient which take responsibility of job management. > FLIP-73 is aimed at > a dedicated job executor which take responsibility of job submission. > However, for full functions to > handle Flink clusters & jobs, Flink cluster management and Flink cluster > deployment still require > public interface.These interface is mainly used for downstream project > developer instead of users > who are only interested in Flink job. > > Further, we already used ClusterClient and ClusterDescriptor in CliFrontend, > sql-client, > scala-shell and so on, which can be regarded as downstream project hosted in > Flink repo. > Given this observation, there is no reason we don't expose API for Flink > cluster management > and Flink cluster deployment. > > So here comes two question: > > 1. Do we want to expose multiple level clients as described above? If no, > why? If so, what does > the plan look like? > 2. Does Executor introduced in the ongoing FLIP-73 break multiple level > clients layout > described above? Since ClusterClient already implements functions for job > submission, where > is Executor in the layout above? Does it overlap with existing client concept? > > Best, > tison.
