Hi Team, I have recently faced the issue that the S3 FileSystem read my core-site.xml until it was on the classpath, but later when I tried to add it using the HADOOP_CONF_DIR then the configuration file was not loaded. Filed a jira [1] and created a PR [2] for fixing it.
HadoopUtils.getHadoopConfiguration is the method which considers all the relevant configurations for accessing / loading the hadoop configuration files, so I used it to fix the issue. The downside is that in this method we instantiate the HdfsConfiguration object which requires me to add the hadoop-hdfs-client as a provided dependency. My question for the more experienced folks - would this cause issues for the users? Could we assume that if the hadoop-common is on the classpath then hadoop-hdfs-client is on the classpath as well? Do you see other possible drawbacks or issues with my approach? Thanks, Peter [1] https://issues.apache.org/jira/browse/FLINK-29754 [2] https://github.com/apache/flink/pull/21148
