Hi, Youngwoo In K8S session, the number of TMs depends on how many slots your job needs and the number of slots per task managers (config key: taskmanager.numberOfTaskSlots). In this case,
# of TM = Ceil(total slots need / taskmanager.numberOfTaskSlots) How many your job's topology and parallelism. For streaming SQL, the whole job graph will locate in one slot by default. So, the number of slots would be equal to the parallelism you set. > Is it possible to run pre-spawned TMs for session mode? I'm looking for a way > to scale the computing resources. i.e., # of TM for the jobs. I might not fully understand your problem. Do you mean starting TMs before submitting the job? If that is the case, - You can try the standalone k8s mode. [1] - Warmup the session by submitting some puppet jobs yourselves and submit your job before those TMs idle timeout. - In FLINK-15959, we will introduce the min number of slots of the cluster. With this feature, you can configure how many TMs needed before submitting the jobs. [1] https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/resource-providers/standalone/kubernetes/ Best, Yangze Guo On Tue, May 11, 2021 at 12:24 PM Youngwoo Kim (김영우) <[email protected]> wrote: > > Hi, > > I have deployed a cluster with session mode on kubernetes and I can see one > deployment, services and one JM. I'm trying to run a SQL query through sql > client. for instance, 'INSERT INTO ... SELECT ...;' > > When I run the query in cli, the Flink session is spinning up a TM for the > query and then the query is running in a job. > > Now, I'm curious. How does Flink calculate the number of TMs for the query? > and also, Is it possible to run pre-spawned TMs for session mode? I'm looking > for a way to scale the computing resources. i.e., # of TM for the jobs. > > Thanks, > Youngwoo
