Hi all, I agree with Xintong's comment: Reducing the number of deployment modes would help users. There is a clearer distinction between session mode and the two other deployment modes (i.e. application and job mode). The difference between application and job mode is not that easy to grasp for newcomers, I imagine. It would also help cleaning up some job-mode-specific code segments in the source code.
It would be interesting to see whether there are other use-cases that are missed in the Application mode (besides the ones already addressed by Biao). I would second Xintong's proposal of deprecating the job-mode rather soonish making users aware of the plans around that deployment mode. That might help encourage users to speak up in case they are not able to find a solution to work around deprecation warnings. I also agree with Xintong's assessment that dropping it should only be done after we're sure that all relevant use cases are met also by other deployment modes considering that (based on the comments above) it is a widely used deployment mode. Matthias On Mon, Jan 24, 2022 at 10:00 AM Xintong Song <tonysong...@gmail.com> wrote: > Sorry for joining the discussion late. > > I'm leaning towards deprecating the per-job mode soonish, and eventually > dropping it in the long-term. > - One less deployment mode makes it easier for users (especially > newcomers) to understand. Deprecating the per-job mode sends the signal > that it is legacy, not recommended, and in most cases users do not need to > care about it. > - For most (if not all) user demands that are satisfied by the per-job > mode but not by the application mode, AFAICS, they can be either workaround > or eventually addressed by the application mode. E.g., make application > mode support shipping local dependencies. > - I'm not sure about dropping the per-job mode soonish, as many users are > still working with it. We'd better not force these users to migrate to the > application mode when upgrading the Flink version. > > Thank you~ > > Xintong Song > > > > On Fri, Jan 21, 2022 at 4:30 PM Konstantin Knauf <kna...@apache.org> > wrote: > >> Thanks Thomas & Biao for your feedback. >> >> Any additional opinions on how we should proceed with per job-mode? As >> you might have guessed, I am leaning towards proposing to deprecate per-job >> mode. >> >> On Thu, Jan 13, 2022 at 5:11 PM Thomas Weise <t...@apache.org> wrote: >> >>> Regarding session mode: >>> >>> ## Session Mode >>> * main() method executed in client >>> >>> Session mode also supports execution of the main method on Jobmanager >>> with submission through REST API. That's how Flinkk k8s operators like >>> [1] work. It's actually an important capability because it allows for >>> allocation of the cluster resources prior to taking down the previous >>> job during upgrade when the goal is optimization for availability. >>> >>> Thanks, >>> Thomas >>> >>> [1] https://github.com/lyft/flinkk8soperator >>> >>> On Thu, Jan 13, 2022 at 12:32 AM Konstantin Knauf <kna...@apache.org> >>> wrote: >>> > >>> > Hi everyone, >>> > >>> > I would like to discuss and understand if the benefits of having >>> Per-Job >>> > Mode in Apache Flink outweigh its drawbacks. >>> > >>> > >>> > *# Background: Flink's Deployment Modes* >>> > Flink currently has three deployment modes. They differ in the >>> following >>> > dimensions: >>> > * main() method executed on Jobmanager or Client >>> > * dependencies shipped by client or bundled with all nodes >>> > * number of jobs per cluster & relationship between job and cluster >>> > lifecycle* (supported resource providers) >>> > >>> > ## Application Mode >>> > * main() method executed on Jobmanager >>> > * dependencies already need to be available on all nodes >>> > * dedicated cluster for all jobs executed from the same main()-method >>> > (Note: applications with more than one job, currently still significant >>> > limitations like missing high-availability). Technically, a session >>> cluster >>> > dedicated to all jobs submitted from the same main() method. >>> > * supported by standalone, native kubernetes, YARN >>> > >>> > ## Session Mode >>> > * main() method executed in client >>> > * dependencies are distributed from and by the client to all nodes >>> > * cluster is shared by multiple jobs submitted from different clients, >>> > independent lifecycle >>> > * supported by standalone, Native Kubernetes, YARN >>> > >>> > ## Per-Job Mode >>> > * main() method executed in client >>> > * dependencies are distributed from and by the client to all nodes >>> > * dedicated cluster for a single job >>> > * supported by YARN only >>> > >>> > >>> > *# Reasons to Keep** There are use cases where you might need the >>> > combination of a single job per cluster, but main() method execution >>> in the >>> > client. This combination is only supported by per-job mode. >>> > * It currently exists. Existing users will need to migrate to either >>> > session or application mode. >>> > >>> > >>> > *# Reasons to Drop** With Per-Job Mode and Application Mode we have two >>> > modes that for most users probably do the same thing. Specifically, for >>> > those users that don't care where the main() method is executed and >>> want to >>> > submit a single job per cluster. Having two ways to do the same thing >>> is >>> > confusing. >>> > * Per-Job Mode is only supported by YARN anyway. If we keep it, we >>> should >>> > work towards support in Kubernetes and Standalone, too, to reduce >>> special >>> > casing. >>> > * Dropping per-job mode would reduce complexity in the code and allow >>> us to >>> > dedicate more resources to the other two deployment modes. >>> > * I believe with session mode and application mode we have to easily >>> > distinguishable and understandable deployment modes that cover Flink's >>> use >>> > cases: >>> > * session mode: olap-style, interactive jobs/queries, short lived >>> batch >>> > jobs, very small jobs, traditional cluster-centric deployment mode >>> (fits >>> > the "Hadoop world") >>> > * application mode: long-running streaming jobs, large scale & >>> > heterogenous jobs (resource isolation!), application-centric deployment >>> > mode (fits the "Kubernetes world") >>> > >>> > >>> > *# Call to Action* >>> > * Do you use per-job mode? If so, why & would you be able to migrate >>> to one >>> > of the other methods? >>> > * Am I missing any pros/cons? >>> > * Are you in favor of dropping per-job mode midterm? >>> > >>> > Cheers and thank you, >>> > >>> > Konstantin >>> > >>> > -- >>> > >>> > Konstantin Knauf >>> > >>> > https://twitter.com/snntrable >>> > >>> > https://github.com/knaufk >>> >> >> >> -- >> >> Konstantin Knauf >> >> https://twitter.com/snntrable >> >> https://github.com/knaufk >> >
