How can I add the package to the flink job or check if it is there? From: Shammon FY <zjur...@gmail.com> Sent: Thursday, May 4, 2023 9:59 PM To: DEROCCO, CHRISTOPHER <cd9...@att.com> Cc: user@flink.apache.org Subject: Re: MSI Auth to Azure Storage Account with Flink Apache Operator not working
Hi DEROCCO, I think you need to check whether there is a hadoop-azure jar file in the classpath of your flink job. From an error message 'Caused by: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.azurebfs.oauth2.MsiTokenProvider not found.', your flink job may be missing this package. Best, Shammon FY On Fri, May 5, 2023 at 4:40 AM DEROCCO, CHRISTOPHER <cd9...@att.com<mailto:cd9...@att.com>> wrote: I receive the error: Caused by: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.azurebfs.oauth2.MsiTokenProvider not found. I’m using flink 1.16 running in Azure Kubernetes using the Flink Apache Kubernetes Operator. I have the following specified in the spec.flinkConfiguration: as per the Apache Kubernetes operator documentation. fs.azure.createRemoteFileSystemDuringInitialization: "true" fs.azure.account.auth.type.storageaccountname.dfs.core.windows.net<https://urldefense.com/v3/__http:/fs.azure.account.auth.type.storageaccountname.dfs.core.windows.net__;!!BhdT!nslIUVS9K-rzMRvjMFpWqBpcsAIiVPAfG6uroDOiSSQfmARHAQCYweWSe-TmKGHGzKD4HpyjvOZFkA$>: OAuth fs.azure.account.oauth.provider.type.<storageaccountname>.dfs.core.windows.net<https://urldefense.com/v3/__http:/dfs.core.windows.net__;!!BhdT!nslIUVS9K-rzMRvjMFpWqBpcsAIiVPAfG6uroDOiSSQfmARHAQCYweWSe-TmKGHGzKD4Hpycm9yrUw$>: org.apache.hadoop.fs.azurebfs.oauth2.MsiTokenProvider fs.azure.account.oauth2.msi.tenant. <storageaccountname>.dfs.core.windows.net<https://urldefense.com/v3/__http:/dfs.core.windows.net__;!!BhdT!nslIUVS9K-rzMRvjMFpWqBpcsAIiVPAfG6uroDOiSSQfmARHAQCYweWSe-TmKGHGzKD4Hpycm9yrUw$>: <MY TENANT ID> fs.azure.account.oauth2.client.id<https://urldefense.com/v3/__http:/fs.azure.account.oauth2.client.id__;!!BhdT!nslIUVS9K-rzMRvjMFpWqBpcsAIiVPAfG6uroDOiSSQfmARHAQCYweWSe-TmKGHGzKD4HpwRB0LkWg$>. <storageaccountname>.dfs.core.windows.net<https://urldefense.com/v3/__http:/dfs.core.windows.net__;!!BhdT!nslIUVS9K-rzMRvjMFpWqBpcsAIiVPAfG6uroDOiSSQfmARHAQCYweWSe-TmKGHGzKD4Hpycm9yrUw$>: <MY CLIENT ID of VM> fs.azure.account.oauth2.client.endpoint. <storageaccountname>.dfs.core.windows.net<https://urldefense.com/v3/__http:/dfs.core.windows.net__;!!BhdT!nslIUVS9K-rzMRvjMFpWqBpcsAIiVPAfG6uroDOiSSQfmARHAQCYweWSe-TmKGHGzKD4Hpycm9yrUw$>: https://login.microsoftonline.com/<https://urldefense.com/v3/__https:/login.microsoftonline.com/__;!!BhdT!nslIUVS9K-rzMRvjMFpWqBpcsAIiVPAfG6uroDOiSSQfmARHAQCYweWSe-TmKGHGzKD4HpzeWh7XLg$><MY TENANT ID>/oauth2/token I also have this specified in the container environment variables. - name: ENABLE_BUILT_IN_PLUGINS value: flink-azure-fs-hadoop-1.16.1.jar I think I’m missing a configuration step because the MsiTokenProvider class is not found based on the logs. Any help would be appreciated. Chris deRocco Senior – Cybersecurity Chief Security Office | STORM Threat Analytics AT&T Middletown, NJ Phone: 732-639-9342 Email: cd9...@att.com<mailto:cd9...@att.com> [cid:image001.png@01D97F3B.3DDB98E0]
