[
https://issues.apache.org/jira/browse/FLINK-6949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17270954#comment-17270954
]
Mikhail Pryakhin commented on FLINK-6949:
-----------------------------------------
Hey [~Tang Yan]
cc [~trohrmann]
I confirm this option doesn't work as expected
I implemented a Dummy mapper that prints out all the resources available on its
classpath, see the code below.
I created a local dir config and fed it with files that I expect to have on
the TMs and JMs classpaths
I submitted a flink job with the following command
{code:java}
flink-1.12.1/bin/flink run -c "org.apache.flink.StreamingJob" -m yarn-cluster
yarn-ship-test-1.0-SNAPSHOT.jar -yt config
{code}
where -yt option ships files in the specified directory (t for transfer)
according to the official doc
But unfortunately all the resources contained in the config folder were not
listed among available resources on the classpth.
I'll implement a unit test.
{code:java}
public class StreamingJob {
public static void main(String[] args) {
try {
StreamExecutionEnvironment env =
StreamExecutionEnvironment.getExecutionEnvironment();
env.fromCollection(Arrays.asList("1", "2", "3", "4"))
.map(new DummyMapper())
.print();
env.execute();
} catch (Exception e) {
e.printStackTrace(System.err);
}
}
public static class DummyMapper extends RichMapFunction<String, String> {
@Override
public void open(Configuration parameters) throws Exception {
System.out.println("Resources on the classpath:");
Arrays.stream(((URLClassLoader)
getClass().getClassLoader()).getURLs()).forEach(System.out::println);
}
@Override
public String map(String value) throws Exception {
return value;
}
}
}
{code}
> Add ability to ship custom resource files to YARN cluster
> ---------------------------------------------------------
>
> Key: FLINK-6949
> URL: https://issues.apache.org/jira/browse/FLINK-6949
> Project: Flink
> Issue Type: Improvement
> Components: Command Line Client, Deployment / YARN
> Affects Versions: 1.3.0
> Reporter: Mikhail Pryakhin
> Assignee: Mikhail Pryakhin
> Priority: Major
>
> *The problem:*
> When deploying a flink job on YARN it is not possible to specify custom
> resource files to be shipped to YARN cluster.
>
> *The use case description:*
> When running a flink job on multiple environments it becomes necessary to
> pass environment-related configuration files to the job's runtime. It can be
> accomplished by packaging configuration files within the job's jar. But
> having tens of different environments one can easily end up packaging as many
> jars as there are environments. It would be great to have an ability to
> separate configuration files from the job artifacts.
>
> *The possible solution:*
> add the --yarnship-files option to flink cli to specify files that should be
> shipped to the YARN cluster.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
