Hi Georg,
I think there is a conceptual misunderstanding. If you reuse the cluster
for several jobs, they need to share the JVM_ARGS since it's the same
process. [1] On Spark, new processes are spawned for each stage afaik.
However, the current recommendation is to use only one ad-hoc cluster per
job/application (which is closer to how Spark works). So if you use YARN,
every job/application spawns a new cluster that just has the right size for
it. Then you can supply new parameters for new YARN submission with
flink run -m yarn-cluster -yD
env.java.opts="-Dconfig.file='config/jobs/twitter-analysis.conf'" \
-class com.github.geoheil.streamingreference.tweets.TweetsAnalysis \
"usecases/tweets/build/libs/tweets_${SCALA_VERSION}-${VERSION}-all.jar"
However, make sure that the path is accessible from within your YARN
cluster, since the driver is probably executed on the cluster (not 100%
sure).
If you want per job level configurations on a shared cluster, I'd recommend
to use normal parameters and initialize PureConfig manually (haven't used
it, so not sure how). Then, you'd probably invoke your program as follows.
flink run \
-class com.github.geoheil.streamingreference.tweets.TweetsAnalysis \
"usecases/tweets/build/libs/tweets_${SCALA_VERSION}-${VERSION}-all.jar"
config.file='config/jobs/twitter-analysis.conf'
For local execution, I had some trouble configuring it as well (tried it
with your code). The issue is that all parameters that we previously tried
are only passed to newly spawned processes while your code is directly
executed in the CLI.
FLINK_ENV_JAVA_OPTS=-Dconfig.file="`pwd`/config/jobs/twitter-analysis.conf"
flink run
-class com.github.geoheil.streamingreference.tweets.TweetsAnalysis \
"usecases/tweets/build/libs/tweets_${SCALA_VERSION}-${VERSION}-all.jar"
FLINK_ENV_JAVA_OPTS is usually parsed from flink-conf.yaml using the
env.java.opts but doesn't respect -Denv.java.opts. I'm not sure if this is
intentional.
If you could put the env.java.opts in the flink-conf.yaml, it would most
likely work for both YARN and local. With FLINK_CONF_DIR you can set a
different conf dir per job. Alternatively, you could also specify both
FLINK_ENV_JAVA_OPTS and -yD to inject the property.
[1] https://stackoverflow.com/a/33855802/10299342
On Thu, Jun 25, 2020 at 12:49 PM Georg Heiler <georg.kf.hei...@gmail.com>
wrote:
> Hi,
>
> but how can I change/configure it per submitted job and not for the whole
> cluster?
>
> Best,
> Georg
>
> Am Do., 25. Juni 2020 um 10:07 Uhr schrieb Arvid Heise <
> ar...@ververica.com>:
>
>> Hi Georg,
>>
>> thank you for your detailed explanation. You want to use
>> env.java.opts[1]. There are flavors if you only want to make it available
>> on job manager or task manager but I guess the basic form is good enough
>> for you.
>>
>> [1]
>> https://ci.apache.org/projects/flink/flink-docs-stable/ops/config.html#jvm-and-logging-options
>>
>> On Wed, Jun 24, 2020 at 10:52 PM Georg Heiler <georg.kf.hei...@gmail.com>
>> wrote:
>>
>>> Hi Arvid,
>>>
>>> thanks for the quick reply. I have a strong Apache spark background.
>>> There, when executing on YARN or locally usually, the cluster is created
>>> on-demand for the duration of the batch /streaming job.
>>> There, there is only the concept of A) master/driver (application
>>> master) B) slave/executor C) Driver: the node where the main class is
>>> invoked. In Sparks`notion, I want the -D parameter to be available on the
>>> (C) Driver node. When translating this to Flink, I want this to be
>>> available to the Main class which is invoked when the job is
>>> submitted/started by the job manager (which should be equivalent to the
>>> driver).
>>>
>>> But maybe my understanding of Flink is not 100% correct yet.
>>>
>>> Unfortunately, using -D directly is not working.
>>>
>>> Best,
>>> Georg
>>>
>>> Am Mi., 24. Juni 2020 um 22:13 Uhr schrieb Arvid Heise <
>>> ar...@ververica.com>:
>>>
>>>> Hi Georg,
>>>>
>>>> could you check if simply using -D is working as described here [1].
>>>>
>>>> If not, could you please be more precise: do you want the parameter to
>>>> be passed to the driver, the job manager, or the task managers?
>>>>
>>>> [1]
>>>> https://ci.apache.org/projects/flink/flink-docs-master/ops/cli.html#deployment-targets
>>>>
>>>> On Wed, Jun 24, 2020 at 8:55 PM Georg Heiler <georg.kf.hei...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> how can I pass additional configuration parameters like spark`s
>>>>> extraJavaOptions to a flink job?
>>>>>
>>>>>
>>>>> https://stackoverflow.com/questions/62562153/apache-flink-and-pureconfig-passing-java-properties-on-job-startup
>>>>>
>>>>> contains the details. But the gist is:
>>>>> flink run --class
>>>>> com.github.geoheil.streamingreference.tweets.TweetsAnalysis \
>>>>> "usecases/tweets/build/libs/tweets_${SCALA_VERSION}-${VERSION}-all.jar
>>>>> " \
>>>>> -yD env.java.opts="-Dconfig.file='config/jobs/twitter-analysis.conf'"
>>>>>
>>>>> is not passing the -Dconfig.file to the flink job!
>>>>>
>>>>> Best,
>>>>> Georg
>>>>>
>>>>
>>>>
>>>> --
>>>>
>>>> Arvid Heise | Senior Java Developer
>>>>
>>>> <https://www.ververica.com/>
>>>>
>>>> Follow us @VervericaData
>>>>
>>>> --
>>>>
>>>> Join Flink Forward <https://flink-forward.org/> - The Apache Flink
>>>> Conference
>>>>
>>>> Stream Processing | Event Driven | Real Time
>>>>
>>>> --
>>>>
>>>> Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany
>>>>
>>>> --
>>>> Ververica GmbH
>>>> Registered at Amtsgericht Charlottenburg: HRB 158244 B
>>>> Managing Directors: Timothy Alexander Steinert, Yip Park Tung Jason, Ji
>>>> (Toni) Cheng
>>>>
>>>
>>
>> --
>>
>> Arvid Heise | Senior Java Developer
>>
>> <https://www.ververica.com/>
>>
>> Follow us @VervericaData
>>
>> --
>>
>> Join Flink Forward <https://flink-forward.org/> - The Apache Flink
>> Conference
>>
>> Stream Processing | Event Driven | Real Time
>>
>> --
>>
>> Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany
>>
>> --
>> Ververica GmbH
>> Registered at Amtsgericht Charlottenburg: HRB 158244 B
>> Managing Directors: Timothy Alexander Steinert, Yip Park Tung Jason, Ji
>> (Toni) Cheng
>>
>
--
Arvid Heise | Senior Java Developer
<https://www.ververica.com/>
Follow us @VervericaData
--
Join Flink Forward <https://flink-forward.org/> - The Apache Flink
Conference
Stream Processing | Event Driven | Real Time
--
Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany
--
Ververica GmbH
Registered at Amtsgericht Charlottenburg: HRB 158244 B
Managing Directors: Timothy Alexander Steinert, Yip Park Tung Jason, Ji
(Toni) Cheng
