Hi Yangze,

Thank you. That's what I'm looking for.

And FLINK-15959 is similar to my given requirements. I hope to run queries
'interactively' but if there are insufficient slots, time for spinning up
new TMs is needed.

Thanks,
Youngwoo

On Tue, May 11, 2021 at 3:36 PM Yangze Guo <[email protected]> wrote:

> Hi, Youngwoo
>
> In K8S session, the number of TMs depends on how many slots your job
> needs and the number of slots per task managers (config key:
> taskmanager.numberOfTaskSlots). In this case,
>
> # of TM  = Ceil(total slots need / taskmanager.numberOfTaskSlots)
>
> How many your job's topology and parallelism. For streaming SQL, the
> whole job graph will locate in one slot by default. So, the number of
> slots would be equal to the parallelism you set.
>
> > Is it possible to run pre-spawned TMs for session mode? I'm looking for
> a way to scale the computing resources. i.e., # of TM for the jobs.
>
> I might not fully understand your problem. Do you mean starting TMs
> before submitting the job? If that is the case,
> - You can try the standalone k8s mode. [1]
> - Warmup the session by submitting some puppet jobs yourselves and
> submit your job before those TMs idle timeout.
> - In FLINK-15959, we will introduce the min number of slots of the
> cluster. With this feature, you can configure how many TMs needed
> before submitting the jobs.
>
> [1]
> https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/resource-providers/standalone/kubernetes/
>
> Best,
> Yangze Guo
>
> On Tue, May 11, 2021 at 12:24 PM Youngwoo Kim (김영우) <[email protected]>
> wrote:
> >
> > Hi,
> >
> > I have deployed a cluster with session mode on kubernetes and I can see
> one deployment, services and one JM. I'm trying to run a SQL query through
> sql client. for instance, 'INSERT INTO ... SELECT ...;'
> >
> > When I run the query in cli, the Flink session is spinning up a TM for
> the query and then the query is running in a job.
> >
> > Now, I'm curious. How does Flink calculate the number of TMs for the
> query? and also, Is it possible to run pre-spawned TMs for session mode?
> I'm looking for a way to scale the computing resources. i.e., # of TM for
> the jobs.
> >
> > Thanks,
> > Youngwoo
>

Reply via email to