Re: Add custom configuration files to TMs classpath on YARN

What we internally did is to inherit from AbstractYarnClusterDescriptor and to customize from there. It's not too difficult but it's nice to see it can be taken care of by AbstractYarnCusterDescriptor. ~Haohui On Wed, Sep 27, 2017 at 9:36 AM Mikhail Pryakhin wrote: > Hi Robert, > > Thanks for

Re: Add custom configuration files to TMs classpath on YARN

Hi Robert, Thanks for your reply! >I believe you should be able to load non-class files through the classloader >as well. Could you please clarify what you mean by this? >Did you see any code that excludes non-class files? No I didn't, but I did see the following code here [1]: if (shipFile.i

Re: Add custom configuration files to TMs classpath on YARN

Hi Mike, For using the DistributedCache approach, you need to have HDFS or another distributed FS available to distribute the files. I would actually like to understand why you said " then this file is copied to the yarn cluster and added to JVM class [...] but is ignored by TM JVM as it is neit

Re: Add custom configuration files to TMs classpath on YARN

Hi Nico, Thanks a lot for you help, but unfortunately, the workaround you suggested doesn't work for me. I tried to leverage the StreamExecutionEnvironment#registerCachedFile method but failed because this instance is created when the application master has already been started therefore the cl

Re: Add custom configuration files to TMs classpath on YARN

Hi Nico! Sounds great, will give it a try and return back with results soon. Thank you so much for your help!! Kind Regards, Mike Pryakhin > On 21 Jun 2017, at 16:36, Nico Kruber wrote: > > A workaround may be to use the DistributedCache. It apparently is not > documented much but the JavaDoc

Re: Add custom configuration files to TMs classpath on YARN

A workaround may be to use the DistributedCache. It apparently is not documented much but the JavaDoc mentions roughly how to use it: https://github.com/apache/flink/blob/master/flink-java/src/main/java/org/apache/ flink/api/java/ExecutionEnvironment.java#L954 /** * Registers a file at the dist

Re: Add custom configuration files to TMs classpath on YARN

Hi guys, any news? I’ve created a jira-ticket https://issues.apache.org/jira/browse/FLINK-6949 . Kind Regards, Mike Pryakhin > On 16 Jun 2017, at 16:35, Mikhail Pryakhin wrote: > > Hi all, > > I run my flink job on yarn cluster and need to

Add custom configuration files to TMs classpath on YARN

Hi all, I run my flink job on yarn cluster and need to supply job configuration parameters via configuration file alongside with the job jar. (configuration file can't be packaged into jobs jar file). I tried to put the configuration file into the folder that is passed via --yarnship option to