Re: adding core-site xml to flink1.11

2020-10-28 Thread Shachar Carmeli
10x On 2020/10/27 10:42:40, Robert Metzger wrote: > Hi, > > it seems that this is what you have to do for now. However, I see that it > would be nice if Flink would allow reading from multiple configuration > files, so that you can have a "common configuration" and a "per cluster" > configurati

Re: adding core-site xml to flink1.11

2020-10-27 Thread Robert Metzger
Hi, it seems that this is what you have to do for now. However, I see that it would be nice if Flink would allow reading from multiple configuration files, so that you can have a "common configuration" and a "per cluster" configuration. I filed a JIRA ticket for a feature request: https://issues.

Re: adding core-site xml to flink1.11

2020-10-27 Thread Shachar Carmeli
Hi, Thank you for your reply, WE are deploying on kubernetes and the xml is part of the common config map to all flink jobs we have(or at least was for previous versions) This means that we need to duplicate the configuration in the flink-conf.yaml for each job instead of having a common confi

Re: adding core-site xml to flink1.11

2020-10-27 Thread Robert Metzger
Hi Shachar, Why do you want to use the core-site.xml to configure the file system? Since we are adding the file systems as plugins, their initialization is customized. It might be the case that we are intentionally ignoring xml configurations from the classpath. You can configure the filesystem i

adding core-site xml to flink1.11

2020-10-24 Thread Shachar Carmeli
Hi, I'm trying to define filesystem to flink 1.11 using core-site.xml I tried adding in the flink-conf.yaml env.hadoop.conf.dir and I see it is added to the classpath also adding environment variable HADOOP_CONF_DIR didn't help The flink 1.11.2 is running on docker using kubernetes I added hado