Thank you Zhanghao, I have some questions according to your comments.
My comments are inlined… Best regards, Burcu From: Zhanghao Chen [mailto:zhanghao.c...@outlook.com] Sent: Tuesday, April 5, 2022 10:33 AM To: Burcu Gul POLAT EGRI <be...@sdt.com.tr>; user@flink.apache.org Subject: [*Newsletter*] Re: Submit jobs via Rest API and deploy Flink on a running Kubernetes cluster (Native way) Hi Burcu, 1. No matter which deployment mode you are using, to submit a Job via Flink REST API, you'll need to firstly set the Flink REST server up, and that essentially means set up the Flink JM ahead of time. [Burcu] Do you mean a running Flink container which is started as jobmanager is enough for REST server? Is this execution like a session mode? Because as far as I understand from Flink documentation samples, after executing jobs in application mode deployment all the pods(JM and TMs) are deleted automatically. How about does Flink treat after executing jobs which are send via Rest API? 1. For submitting job via Flink REST API, you could refer to https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/ops/rest_api/#jars-jarid-run. However, only savepoint-related Flink configs are supported by this API so far. [Burcu] Yes, I have already used this api, and I am trying to learn the usage of query parameters of this end point. I mean, are there any way to set parameters like “program-args”, esp. below yellow marked one? (I assume that the parameters start with –D can be send as “program-args”) CLI command ./bin/flink run-application --target kubernetes-application -Dkubernetes.cluster-id=my-first-application-cluster -Dkubernetes.container.image=k8s-harbor.telespazio.corp/sdt-dproc-flink-test/custom-flink -Dkubernetes.service-account=flink-appmode-service-account -Dkubernetes.rest-service.exposed.type=NodePort -Dkubernetes.namespace=sdt-dproc-flink-test-app -Dkubernetes.config.file=/home/devuser/.kube/config local:///opt/flink/usrlib/my-flink-job.jar what is the rest equivalent of above job? REST API | Apache Flink<https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/ops/rest_api/#jars-jarid-run> REST API # Flink has a monitoring API that can be used to query status and statistics of running jobs, as well as recent completed jobs. This monitoring API is used by Flink’s own dashboard, but is designed to be used also by custom monitoring tools. The monitoring API is a REST-ful API that accepts HTTP requests and responds with JSON data. nightlies.apache.org 1. I'm supposing your goal is to develop a service for other users to submit their jobs via RESTful API. If that's the case, you could consider developing a new service that submits Flink job via Flink command line interface or Flink client Java API and provides a RESTful API to users. You could also checkout the newly released Flink Kubernetes Operator (not sure if the doc is ready yet, you can search that in the mailing list) to see if it could be integrated it into your service. [Burcu] More or less yes, I am implementing a microservice that is responsible to execute data processing workflows which are defined by operator users and the workflow can contain Flink related steps. Whenever the workflow execution reaches a Flink related step, it sends a job to Flink to execute job on task managers. I thought that sending jobs via Flink REST API is more suitable for my microservice rather than calling from commandline. Could you provide a sample or documentation of Flink client Java API? I have already tried to check Flink Kubernetes Operator but it requires more Kubernetes knowledge to understand documentation. But I will try it in parallel again. Thank you very much again…… Best, Zhanghao Chen ________________________________ From: Burcu Gul POLAT EGRI <be...@sdt.com.tr<mailto:be...@sdt.com.tr>> Sent: Monday, April 4, 2022 16:16 To: user@flink.apache.org<mailto:user@flink.apache.org> <user@flink.apache.org<mailto:user@flink.apache.org>> Subject: Submit jobs via Rest API and deploy Flink on a running Kubernetes cluster (Native way) Dear all, I am trying to implement a Rest client for Flink to send jobs via Restful Flink services. And also I want to integrate Flink and Kubernetes natively. I have decided to use “Application Mode” as deployment mode according to Flink documentation . I have already implemented a job and packaged it as jar. And I have tested it on Standalone Flink. But my aim is to move on Kubernetes and deploy my application in Application mode via Rest API of Flink. I have already investigated the samples at Flink documentation - Native Kubernetes<https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/deployment/resource-providers/native_kubernetes/#native-kubernetes>. But I cannot find a sample for executing same samples via Restful services (esp. how to set --target kubernetes-application/kubernetes-session or other parameters). In addition to samples, I checked out the Flink sources from GitHub and tried to find some sample implementation or get some clue. I think the below ones are related with my case. - org.apache.flink.client.program.rest. RestClusterClient - org.apache.flink.kubernetes. KubernetesClusterDescriptorTest. testDeployApplicationCluster But they are all so complicated for me to understand below points. - For application mode, are there any need to initialize a container to serve Flink Rest services before submitting job? If so, is it JobManager? - For application mode, how can I set the same command line parameters via Rest services? - For session mode, in command line samples, kubernetes-session.sh is executed before job submission to initialize a JobManager container. How sould I do this step via Rest client? - For session mode, how can I set the same command line parameters via Rest services? Although the command line samples send .jar job as parameter, should I upload jar before submitting job? Could you please provide me some clue/sample to continue my implementation? Best regards, Burcu Bu e-posta ve içeriği kişiye özel ve gizli bilgiler içerebilir. Eğer mesajın muhatabı veya muhataba iletmekle yükümlü yetkili temsilcisi siz değilseniz, bu mesajı çoğaltmak, dağıtmak, açıklamak dahil olmak üzere herhangi bir suretle kullanmamanız gerektiğini, aksine davranışınızın hukuka aykırılık teşkil edebileceğini bildiririz. Eğer bu mesajı yanlışlıkla aldıysanız, lütfen göndericiye e-posta ile bildirerek siliniz. Bu mesajda belirtilen şahsi görüşler göndericiye aittir ve SDT A.Ş.’nin resmi görüşünü temsil etmeyebilir. This email and its contents may contain information that is privileged and confidential. If you are not an intended recipient,or the agent responsible for delivering this email to the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited and may be unlawful. If you received this email in error, please notify the sender by replying to this email and delete the email sent in error. Personel opinions presented in this e-mail message are solely those of the author and do not necessarily represent SDT A.S.`s formal and authorized views. Bu e-posta ve içeriği kişiye özel ve gizli bilgiler içerebilir. Eğer mesajın muhatabı veya muhataba iletmekle yükümlü yetkili temsilcisi siz değilseniz, bu mesajı çoğaltmak, dağıtmak, açıklamak dahil olmak üzere herhangi bir suretle kullanmamanız gerektiğini, aksine davranışınızın hukuka aykırılık teşkil edebileceğini bildiririz. Eğer bu mesajı yanlışlıkla aldıysanız, lütfen göndericiye e-posta ile bildirerek siliniz. Bu mesajda belirtilen şahsi görüşler göndericiye aittir ve SDT A.Ş.’nin resmi görüşünü temsil etmeyebilir. This email and its contents may contain information that is privileged and confidential. If you are not an intended recipient,or the agent responsible for delivering this email to the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited and may be unlawful. If you received this email in error, please notify the sender by replying to this email and delete the email sent in error. Personel opinions presented in this e-mail message are solely those of the author and do not necessarily represent SDT A.S.`s formal and authorized views.
