Thanks for the response, Asimansu. I should have been a bit more clearer and shared some additional context on our internal deployment. Currently, we are running *Flink in YARN application mode* for *batch* execution purposes (we also run it for stream execution as well). In the YARN application mode, client only gets the YARN application ID back when the Flink application is submitted, JobManager (as part of Application Master) generates the JobId whenever it constructs the `StreamGraph` and therefore Flink JobId is logged in the JM logs.
Why do we need the Flink Job Id in the Flink Client? - Currently, Flink Job archives are archived by the Flink JobId therefore to show the link of the completed application (Flink UI in HistoryServer) in the orchestrator (eg: Airflow), Flink JobId is required. Airflow currently only has access to the Flink client logs. Hope that helps. Regards Venkata krishnan On Thu, Mar 14, 2024 at 6:22 PM Asimansu Bera <asimansu.b...@gmail.com> wrote: > Hello Venkat, > > There are few ways to get the JobID from the client side. JobID is alpha > numeric as 9eec4d17246b5ff965a43082818a3336. > When you submit the job using flink command line client , Job is returned > as > > Job has been submitted with JobID 9eec4d17246b5ff965a43082818a3336 > > 1. using below command , you get a list of jobs which are running with > jobIDs > $flink job list > > 2. using REST API > http://localhost:8081/jobs/ > <https://urldefense.com/v3/__http://localhost:8081/jobs/__;!!IKRxdwAv5BmarQ!faNgzFF1xX_bGkjezZJn0ijh4LNpDtXxp8qwvk235dV4LbDEfYOtlIagr9tU9JW3X1_04VmCIpKltpdCnynP3ImKJAA$> > > Hope this helps. > > Regards, > Asimansu > > > On Thu, Mar 14, 2024 at 3:11 PM Venkatakrishnan Sowrirajan < > vsowr...@asu.edu> wrote: > >> Junrui, >> >> Thanks for your answer for the above questions. >> >> Allison and I work together on Flink. One of the main questions is, is >> there an easy way to get the Flink "JobID" from the Flink client side? >> Without the "JobID", users have no way to access Flink HistoryServer >> other than searching through the list of applications in FlinkHistoryServer >> UI. Btw, in this case we are running Flink in batch execution mode. >> >> Regards >> Venkat >> >> >> On Wed, Mar 13, 2024 at 7:58 PM Junrui Lee <jrlee....@gmail.com> wrote: >> >>> Hi Allison, >>> >>> The PIPELINE_FIXED_JOB_ID configuration option is not intended for >>> public use. IIUC, the only way to manually specify the jobId is submitting >>> a job through the JAR RUN REST API, where you can provide the jobId in the >>> request body ( >>> https://nightlies.apache.org/flink/flink-docs-master/docs/ops/rest_api/#jars-jarid-run >>> <https://urldefense.com/v3/__https://nightlies.apache.org/flink/flink-docs-master/docs/ops/rest_api/*jars-jarid-run__;Iw!!IKRxdwAv5BmarQ!do5GHfQPWDwTL1BcBk_Y1pMkZvfSHYRldgXzKOpIF1aRWER97Clb1yh7CQaiUbai4ZXDM_edEFOPqZc4o8N5zw$> >>> ). >>> >>> Best, >>> Junrui >>> >>> Allison Chang via user <user@flink.apache.org> 于2024年3月14日周四 08:16写道： >>> >>>> Hi, >>>> >>>> I was wondering if there is any way to manually set the jobID for the >>>> jobGraph. I noticed that there is a configuration for >>>> PIPELINE_FIXED_JOB_ID, but there doesn't seem to be a way to set it via >>>> config with the StreamingJobGraphGenerator.java. Would appreciate any >>>> assistance if anyone has done something similar. >>>> >>>> Best, >>>> >>>> *Allison Chang* >>>> >>>> >>>>
