Thanks Robert for helpful reply. I have follow up on the Q2 - "In general, we recommend running one JobManager per job” I understand how this can be achieved while running in Yarn, I.e. by submitting single Flink Jobs.
Is their some other way of setting Flink to configure single Jobmanager per job ? From: Robert Metzger <rmetz...@apache.org<mailto:rmetz...@apache.org>> Reply-To: "user@flink.apache.org<mailto:user@flink.apache.org>" <user@flink.apache.org<mailto:user@flink.apache.org>> Date: Tuesday, June 21, 2016 at 8:23 AM To: "user@flink.apache.org<mailto:user@flink.apache.org>" <user@flink.apache.org<mailto:user@flink.apache.org>>, Till Rohrmann <trohrm...@apache.org<mailto:trohrm...@apache.org>> Cc: Aljoscha Krettek <aljos...@apache.org<mailto:aljos...@apache.org>> Subject: Re: Documentation for translation of Job graph to Execution graph Hi, the link has been added newly, yes. Regarding Q1, since there is no documentation right now, I have to refer you to our code. In the JobManager.scala class, there is a method "private def submitJob(jobGraph, ...") where the ExecutionGraph is created. I think that's a good starting point for looking through the code. (I also added Till to the message if he wants to chime in) Q2: Currently, Flink doesn't add the job name to the logs, so its indeed not very easy to separate the log entries generated by different jobs. In general, we recommend running one JobManager per job (multiple jobs is of course also supported). On Sat, Jun 18, 2016 at 1:41 AM, Bajaj, Abhinav <abhinav.ba...@here.com<mailto:abhinav.ba...@here.com>> wrote: Hi, Thanks for sharing this link. I have not see it before. May be this is newly added in 1.0 docs. I will go through it. In general, there are two things I am trying to understand and get comfortable with - 1. How a Job graph is translated to Execution graph. The logs and monitoring APIs are for the Execution graph. So, I need to map them to the Job graph. I am trying to bridge this gap. 2. The job manager & task manager logs are tricky to decipher. Especially when there are multiple jobs running. Is there a way to filter the logs for a single job ? ~ Abhi From: Aljoscha Krettek <aljos...@apache.org<mailto:aljos...@apache.org>> Reply-To: "user@flink.apache.org<mailto:user@flink.apache.org>" <user@flink.apache.org<mailto:user@flink.apache.org>> Date: Friday, June 17, 2016 at 2:31 AM To: "user@flink.apache.org<mailto:user@flink.apache.org>" <user@flink.apache.org<mailto:user@flink.apache.org>> Subject: Re: Documentation for translation of Job graph to Execution graph Hi, I'm afraid there is no documentation besides the link that you posted and this one: https://ci.apache.org/projects/flink/flink-docs-release-1.0/concepts/concepts.html. With what parts are you having trouble? Maybe I can help. Cheers, Aljoscha On Thu, 16 Jun 2016 at 19:31 Bajaj, Abhinav <abhinav.ba...@here.com<mailto:abhinav.ba...@here.com>> wrote: Hi, When troubleshooting a flink job, it is tricky to map the Job graph (application code) to the logs & monitoring REST APIs. So, I am trying to find documentation on how a Job graph is translated to Execution graph. I found this - https://ci.apache.org/projects/flink/flink-docs-release-1.0/internals/job_scheduling.html Any detailed documentation on the design and code components will be helpful. Thanks, Abhi
