Github user mxm commented on the pull request:
https://github.com/apache/flink/pull/1214#issuecomment-145949994
@sachingoel0101 You're right. The detached YARN execution mode is broken
for jobs which use eager execution and build other jobs using the results.
However, the detached YARN execution could work if the ApplicationMaster
executed the program. Essentially, this means shifting the Client to the YARN
cluster.
@StephanEwen The way Sachin was going to handle the eager execution during
detached mode is to throw an Exception. However, the current code doesn't
handle all cases. It is in the nature of an interactive session that multiple
jobs can be submitted. Results from submitted jobs can be reused not only via
count/collect but also using the returned JobExecutionResult. Therefore, it is
very difficult to deal with all these cases. IMHO we can only support
non-interactive detached job submissions.
All in all, I think this is a nice idea but implementing it has too much
overhead. The implications of the changes are not quite clear. I pledge to
change this to "Won't Fix" until we think about a proper implementation. In the
meantime, we should fix the detached YARN mode or remove it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
