Hi Shengkai, Today we currently use application mode. It is an option and may be the recommendation.
Specifically for Batch jobs, we have Machine Learning pipelines that are ephemeral however they contain very different dependencies depending on the workload. >From my perspective, Batch jobs work well on Session Clusters. However, due to the differing images you cannot run different workloads on the same session cluster. Making the session cluster essentially useless. Ryan van Huuksloot Sr. Production Engineer | Streaming Platform [image: Shopify] <https://www.shopify.com/?utm_medium=salessignatures&utm_source=hs_email> On Tue, Dec 3, 2024 at 1:20 AM Shengkai Fang <fskm...@gmail.com> wrote: > Hi. > > Why needs different image for taskmanager? Do you mean different operators > require different resources? > > As far as I know, JM supports to manage taskmanager with different > profiles. For example, a cluster may consists of two taskmanagers with > following profiles: > * TM1 contains 4 slots, every slot has 2 core, 4GB Memory > * TM2 contains 4 slots, every slot have 1core, 2GB Memory > > > the scheduler would need some level of job isolation > > You can use application mode to run the job. In application mode, the > cluster is dedicated for the job. > > Best, > Shengkai >
