[
https://issues.apache.org/jira/browse/FLINK-4114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15353046#comment-15353046
]
Chris Hogue commented on FLINK-4114:
------------------------------------
I should add that since filing this I realized I missed a parameter on the yarn
integration that we can use to accomplish this. The -ynm param allows us to
name the yarn application. Then along with the yarn application -list command
we can find the application ID associated with a named job. It's possibly
slightly brittle as we end up grep'ing and awk'ing out the application ID from
the list command, but it should be functional.
If you want to continue with this JIRA to provide a more direct integration
that's great, just wanted to mention that we do seem to have a way to do this
for now. It'd be fine if you want to lower the priority of this IMO.
> Need a way to manage multiple named, long-lived jobs on a single YARN cluster
> in an automated manner
> ----------------------------------------------------------------------------------------------------
>
> Key: FLINK-4114
> URL: https://issues.apache.org/jira/browse/FLINK-4114
> Project: Flink
> Issue Type: Improvement
> Components: YARN Client
> Reporter: Chris Hogue
>
> We are running several Flink jobs on a single YARN cluster. Currently each
> Flink job is run in its own YARN session (and thus its own YARN application
> ID). The difficulty comes in that we want to manage each of these jobs
> individually by name. For example we want to start, stop, update one job
> without affecting others. The primary access to these jobs is via the YARN
> application ID, which is not meaningful to discern which flink job it is
> running.
> It would be nice if we had tools that would allow us to manage the flink jobs
> by name and have it do the right thing with the YARN session. Today we can
> use 'flink run' and have it start a YARN session for that job, but from that
> point forward we have only the YARN application ID to work with.
> As a concrete example suppose we have 2 jobs with names JobA and JobB. We'd
> want a way to so something like:
> flink run <JobA jar>; flink run <JobB jar>
> We'd then want to be able to call:
> flink cancel JobA
> The cancel command would spin down the YARN session for JobA in addition to
> the flink job, leaving JobB running as normal. I've simplified the commands
> leaving out other options for illustrative purposes. And we'll want to be
> able to use savepoints through these steps as well.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
