Hi Ryan. Thanks for your input. I am not a k8s expert, but I know that Flink k8s deployments supports to get Flink TaskManager with specified pod template[1], which supports to specify image. @Junrui may provide more detailed information about this topic.
If different taskmanager has different workload, it means the slot in the different taskamanger has different profiles. Otherwise, scheduler doesn't know the difference among different slots and may choose the wrong slot to run the task. I am just curious what's the difference between the ETL job and ML job. Best, Shengkai [1] https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/resource-providers/native_kubernetes/#pod-template Ryan van Huuksloot <ryan.vanhuuksl...@shopify.com.invalid> 于2024年12月3日周二 22:11写道： > Hi Shengkai, > > Today we currently use application mode. It is an option and may be the > recommendation. > > Specifically for Batch jobs, we have Machine Learning pipelines that are > ephemeral however they contain very different dependencies depending on the > workload. > From my perspective, Batch jobs work well on Session Clusters. However, due > to the differing images you cannot run different workloads on the same > session cluster. Making the session cluster essentially useless. > > Ryan van Huuksloot > Sr. Production Engineer | Streaming Platform > [image: Shopify] > <https://www.shopify.com/?utm_medium=salessignatures&utm_source=hs_email> > > > On Tue, Dec 3, 2024 at 1:20 AM Shengkai Fang <fskm...@gmail.com> wrote: > > > Hi. > > > > Why needs different image for taskmanager? Do you mean different > operators > > require different resources? > > > > As far as I know, JM supports to manage taskmanager with different > > profiles. For example, a cluster may consists of two taskmanagers with > > following profiles: > > * TM1 contains 4 slots, every slot has 2 core, 4GB Memory > > * TM2 contains 4 slots, every slot have 1core, 2GB Memory > > > > > the scheduler would need some level of job isolation > > > > You can use application mode to run the job. In application mode, the > > cluster is dedicated for the job. > > > > Best, > > Shengkai > > >
