Do you also want to answer
https://stackoverflow.com/questions/62562153/apache-flink-and-pureconfig-passing-java-properties-on-job-startup
?
Your suggestion seems to work well.
Best,
Georg
Am Do., 25. Juni 2020 um 15:32 Uhr schrieb Arvid Heise :
> You are welcome.
>
> I'm not an expert on the y
You are welcome.
I'm not an expert on the yarn executor but I hope that
-yt,--yarnship Ship files in the specified directory
(t for transfer)
can help [1]. Oddly this option is not given on the YARN page. But it
should be available
Thanks a lot!
Your point is right.
One Cluster per job should be used in the thought model to be comparable.
In particular for YARN:
-yD env.java.opts="-Dconfig.file='config/jobs/twitter-analysis.conf'"
You mentioned, that the path must be accessible. Spark has a --files
parameter and then the
Hi Georg,
I think there is a conceptual misunderstanding. If you reuse the cluster
for several jobs, they need to share the JVM_ARGS since it's the same
process. [1] On Spark, new processes are spawned for each stage afaik.
However, the current recommendation is to use only one ad-hoc cluster per
Hi,
but how can I change/configure it per submitted job and not for the whole
cluster?
Best,
Georg
Am Do., 25. Juni 2020 um 10:07 Uhr schrieb Arvid Heise :
> Hi Georg,
>
> thank you for your detailed explanation. You want to use env.java.opts[1].
> There are flavors if you only want to make it
Hi Georg,
thank you for your detailed explanation. You want to use env.java.opts[1].
There are flavors if you only want to make it available on job manager or
task manager but I guess the basic form is good enough for you.
[1]
https://ci.apache.org/projects/flink/flink-docs-stable/ops/config.html
Hi Arvid,
thanks for the quick reply. I have a strong Apache spark background. There,
when executing on YARN or locally usually, the cluster is created on-demand
for the duration of the batch /streaming job.
There, there is only the concept of A) master/driver (application master)
B) slave/executo
Hi Georg,
could you check if simply using -D is working as described here [1].
If not, could you please be more precise: do you want the parameter to be
passed to the driver, the job manager, or the task managers?
[1]
https://ci.apache.org/projects/flink/flink-docs-master/ops/cli.html#deployment
Hi,
how can I pass additional configuration parameters like spark`s
extraJavaOptions to a flink job?
https://stackoverflow.com/questions/62562153/apache-flink-and-pureconfig-passing-java-properties-on-job-startup
contains the details. But the gist is:
flink run --class
com.github.geoheil.streami
