Hi Raihan, I believe the flink k8s operator has some implicit assumption that each application should contain only one job so that it can manage the lifecycle of the job reasonably. Your usage of deploying multiple jobs in a single python file(i.e. in a single flink application) may be not preferred in this case.
More details why you only see the first job: currently, when flink k8s operator deploys a job, it will try to set the job id of the application (See codes <https://github.com/apache/flink-kubernetes-operator/blob/249a6945aaa847a506c42111d7332ad072578095/flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/reconciler/deployment/ApplicationReconciler.java#L178> for more details). As a result, when you call multiple "executeAsync()", in flink client's execute() implementation <https://github.com/apache/flink/blob/29736b8c01924b7da03d4bcbfd9c812a8e5a08b4/flink-clients/src/main/java/org/apache/flink/client/deployment/application/executors/EmbeddedExecutor.java#L100>, as the PIPELINE_FIXED_JOB_ID is set in the config, all "executeAsync()" will share the same job id. In this case, only the first "executeAsync()" will be executed and the following is a no-op. I think that's why you see only the first job is submitted. In the JM log, you may see messages like `Job 4eadb4b025efa5dc2a6ef2199af6ca2b was recovered successfully.` which implies the following executeAsync() does not take effect. Best, Biao Geng Best, Biao Geng Raihan Sunny <raihan.su...@selisegroup.com> 于2024年5月7日周二 13:48写道： > Hi everyone, > > I need some help with deploying multiple jobs from a single main function > in Application mode using Flink Kubernetes Operator. As per the > documentation [1] it should be possible to use multiple "executeAsync()" to > deploy multiple jobs from the same file. This indeed is the case when > running the job locally using the CLI with something like `/bin/flink run > -pym main -pyfs /project/` (I'm using PyFlink btw) and I can see multiple > jobs running in the UI. However, when I try to deploy the same job using > Flink Kubernetes Operator, it seems that only the first job gets submitted. > The second job is never submitted although the code leading up to > "executeAsync()" does get executed. > > This is a minimal representation of the deployment manifest that I tried > to run: > > apiVersion: flink.apache.org/v1beta1 > kind: FlinkDeployment > metadata: > name: flink-job > spec: > image: flinkimage > imagePullPolicy: IfNotPresent > flinkVersion: v1_17 > flinkConfiguration: > taskmanager.numberOfTaskSlots: "1" > state.savepoints.dir: hdfs://... > state.checkpoints.dir: hdfs://... > serviceAccount: flink > jobManager: > resource: > memory: "1024m" > cpu: 0.5 > taskManager: > resource: > memory: "1024m" > cpu: 0.5 > job: > jarURI: local:///opt/flink/opt/flink-python_2.12-1.17.0.jar > entryClass: "org.apache.flink.client.python.PythonDriver" > args: ["python", "-pym", "main", "-pyfs", "/project/"] > parallelism: 1 > upgradeMode: savepoint > state: running > > Any help would be greatly appreciated. I'm using Flink v1.17 and Flink > Kubernetes Operator v1.7.0. > > [1] > https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/deployment/overview/#application-mode > <https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/deployment/overview/#application-mode),> > > > Thanks, > Sunny >
