Hi Preston, Before FileSystems are accessed, they are first initialized with the configuration coming from the flink-conf. [1] It seems that your local execution in IDE is bypassing that, so I'd call that manually. Which entry point are you using? We should probably fix it, such that it's also calling initialize. Also make sure your config options are prefixed with (fs.)azure. because only they are relayed.
[1] https://github.com/apache/flink/blob/master/flink-core/src/main/java/org/apache/flink/core/fs/FileSystem.java#L319-L319 On Wed, Oct 20, 2021 at 12:30 AM Preston Price <nacro...@gmail.com> wrote: > Some details about my runtime/environment: > Java 11 > Flink version 1.14.0 > Running locally in IntelliJ > > The error message that I am getting is: Configuration property > {storage_account}.dfs.core.windows.net not found. > Reading through all the docs hasn't yielded much help. > > In the Flink docs here > <https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/deployment/filesystems/azure/#credentials-configuration>, > it's claimed that we can set the credentials for ABFS by specifying the > value in flink-conf.yaml, so this is what I am trying. However, in the > code path expressed in the stack trace, I don't believe the configuration > loaded from flink-conf.yaml is ever consulted. Here are the relevant > parts of the stack trace: > Caused by: > org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.azurebfs.contracts.exceptions.ConfigurationPropertyNotFoundException: > Configuration property {storage_account}.dfs.core.windows.net not found. > at > org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.azurebfs.AbfsConfiguration.getStorageAccountKey(AbfsConfiguration.java:372) > ~[flink-azure-fs-hadoop-1.14.0.jar:1.14.0] > at > org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.initializeClient(AzureBlobFileSystemStore.java:1133) > ~[flink-azure-fs-hadoop-1.14.0.jar:1.14.0] > at > org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.<init>(AzureBlobFileSystemStore.java:174) > ~[flink-azure-fs-hadoop-1.14.0.jar:1.14.0] > at > org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.initialize(AzureBlobFileSystem.java:110) > ~[flink-azure-fs-hadoop-1.14.0.jar:1.14.0] > at > org.apache.flink.fs.azurefs.AbstractAzureFSFactory.create(AbstractAzureFSFactory.java:79) > ~[flink-azure-fs-hadoop-1.14.0.jar:1.14.0] > at > org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:508) > ~[flink-core-1.14.0.jar:1.14.0] > at org.apache.flink.core.fs.FileSystem.get(FileSystem.java:409) > ~[flink-core-1.14.0.jar:1.14.0] > > When we first call org.apache.flink.core.fs.FileSystem.get the > FileSystemFactories are initialized with a new/empty configuration > <https://github.com/apache/flink/blob/release-1.14.0/flink-core/src/main/java/org/apache/flink/core/fs/FileSystem.java#L497>, > not the configuration loaded from flink-conf.yaml Therefore, later on > when we're calling AbstractAzureFSFactory.create > <https://github.com/apache/flink/blob/release-1.14.0/flink-filesystems/flink-azure-fs-hadoop/src/main/java/org/apache/flink/fs/azurefs/AbstractAzureFSFactory.java#L74> > we > have an empty config, so the call > to HadoopConfigLoader.getOrLoadHadoopConfig() then > HadoopConfigLoader.loadHadoopConfigFromFlink > <http://HadoopConfigLoader.loadHadoopConfigFromFlinkhttps://github.com/apache/flink/blob/99c2a415e9eeefafacf70762b6f54070f7911ceb/flink-filesystems/flink-hadoop-fs/src/main/java/org/apache/flink/runtime/util/HadoopConfigLoader.java#L95> > can't > merge in our config from flink-conf.yaml > > So if the Configuration loaded from flink-conf.yaml isn't supplied to the > AbstractAzureFSFactory, how do we configure Flink to connect to Azure Data > Lake? > > Thanks! >
