Hi Ryan.

Thanks for your input. I am not a k8s expert, but I know that Flink k8s
deployments supports to get Flink TaskManager with specified pod
template[1], which supports to specify image. @Junrui may provide more
detailed information about this topic.

If different taskmanager has different workload, it means the slot in the
different taskamanger has different profiles. Otherwise, scheduler doesn't
know the difference among different slots and may choose the wrong slot to
run the task. I am just curious what's the difference between the ETL job
and ML job.

Best,
Shengkai

[1]
https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/resource-providers/native_kubernetes/#pod-template

Ryan van Huuksloot <ryan.vanhuuksl...@shopify.com.invalid> 于2024年12月3日周二
22:11写道:

> Hi Shengkai,
>
> Today we currently use application mode. It is an option and may be the
> recommendation.
>
> Specifically for Batch jobs, we have Machine Learning pipelines that are
> ephemeral however they contain very different dependencies depending on the
> workload.
> From my perspective, Batch jobs work well on Session Clusters. However, due
> to the differing images you cannot run different workloads on the same
> session cluster. Making the session cluster essentially useless.
>
> Ryan van Huuksloot
> Sr. Production Engineer | Streaming Platform
> [image: Shopify]
> <https://www.shopify.com/?utm_medium=salessignatures&utm_source=hs_email>
>
>
> On Tue, Dec 3, 2024 at 1:20 AM Shengkai Fang <fskm...@gmail.com> wrote:
>
> > Hi.
> >
> > Why needs different image for taskmanager? Do you mean different
> operators
> > require different resources?
> >
> > As far as I know, JM supports to manage taskmanager with different
> > profiles. For example, a cluster may consists of two taskmanagers with
> > following profiles:
> > * TM1 contains 4 slots, every slot has 2 core, 4GB Memory
> > * TM2 contains 4 slots, every slot have 1core, 2GB Memory
> >
> > > the scheduler would need some level of job isolation
> >
> > You can use application mode to run the job. In application mode, the
> > cluster is dedicated for the job.
> >
> > Best,
> > Shengkai
> >
>

Reply via email to