Thanks Alexis, I will be trying that out today. If it works I will share back and try adding it to the docs.
________________________________ From: Alexis Sarda-Espinosa <sarda.espin...@gmail.com> Sent: Thursday, March 2, 2023 3:33:03 PM To: Ivan Webber <ivan.web...@microsoft.com> Cc: user <user@flink.apache.org> Subject: Re: [EXTERNAL] Re: Secure Azure Credential Configuration You don't often get email from sarda.espin...@gmail.com. Learn why this is important<https://aka.ms/LearnAboutSenderIdentification> Hi Ivan, please always include the whole distribution list since answers may help others as well. I would also think about implementing your own provider(s), but some things I know: - There are 2 different KeyProvider interfaces (which isn't explicitly documented from what I can tell): * org.apache.hadoop.fs.azure.KeyProvider - WASB * org.apache.hadoop.fs.azurebfs.services.KeyProvider - ABFS (I think) - Flink shades the hadoop classes under org.apache.flink.fs.shaded.hadoop3... so you would need to implement your providers against the shaded interfaces. - The documentation for Flink plugins [1] shows an s3 folder with multiple jars, so I imagine you could add a jar with your key providers to a folder with the azure-fs jar, but I've never tested this. However, I believe this whole shading and plugin details are only relevant if you want Flink to access the azure FS for its checkpoints and/or savepoints, if you need to access the FS directly in your code, I imagine you're better off including the relevant hadoop jars in your fat jar without going through Flink's plugin system. This is my impression, but maybe someone else can correct me if I'm wrong. [1] https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/filesystems/plugins/<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fnightlies.apache.org%2Fflink%2Fflink-docs-master%2Fdocs%2Fdeployment%2Ffilesystems%2Fplugins%2F&data=05%7C01%7CIvan.Webber%40microsoft.com%7C302a8781f35f4b64cfaf08db1b767f30%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638133968331853429%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000%7C%7C%7C&sdata=8W361g8DGOeDIipbVIHZC4TyzxFs8XiQIMFoW7zLUbQ%3D&reserved=0> Regards, Alexis. Am Do., 2. März 2023 um 23:46 Uhr schrieb Ivan Webber <ivan.web...@microsoft.com<mailto:ivan.web...@microsoft.com>>: Hello Alexis, I was actually thinking I’d use both WASB and ABFS, but I looked at the source for EnvironmentVariableKeyProvider and it only reads a single specific environment variable where my pipeline actually needs to bring together data stored in different blob and ADLS accounts. I couldn’t find anything about providing my own KeyProvider but I considered trying it as an experiment at one point. From: Alexis Sarda-Espinosa<mailto:sarda.espin...@gmail.com> Sent: Thursday, March 2, 2023 2:38 PM To: Ivan Webber<mailto:ivan.web...@microsoft.com> Cc: user<mailto:user@flink.apache.org> Subject: [EXTERNAL] Re: Secure Azure Credential Configuration You don't often get email from sarda.espin...@gmail.com<mailto:sarda.espin...@gmail.com>. Learn why this is important<https://aka.ms/LearnAboutSenderIdentification> Hi Ivan, Mercy is always free. Are you using WASB or ABFS? I presume it's the latter, since that's the one that can't use EnvironmentVariableKeyProvider, but just to be sure. Regards, Alexis. On Thu, 2 Mar 2023, 23:07 Ivan Webber via user, <user@flink.apache.org<mailto:user@flink.apache.org>> wrote: TLDR: I will buy your coffee if you can help me understand to securely configure Azure credentials (doc page<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fnightlies.apache.org%2Fflink%2Fflink-docs-master%2Fdocs%2Fdeployment%2Ffilesystems%2Fazure%2F&data=05%7C01%7CIvan.Webber%40microsoft.com%7C302a8781f35f4b64cfaf08db1b767f30%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638133968331853429%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000%7C%7C%7C&sdata=C3898nZaxzivVsSaN2jMqME93PR%2BicS%2By9MFJKV%2B%2Fyk%3D&reserved=0> for reference). I am a junior developer tasked with being the first person to learn the Apache Flink framework. I know that storing secrets in flink-conf.yaml in a container is a bad idea. I’ve tried exposing Azure storage keys as env vars and using `config.setString`, but those properties seem to get overridden. I plan on using Flink operator, so if you can show me in that context that’d be ideal. Thanks, and sorry for bothering everyone. I’ve just exhausted myself and am hopeful someone will have mercy for me. I really will Venmo you $5 for coffee if you want. Thanks, Ivan Larger code examples: Setting dynamic properties before executing the job doesn’t work because the values seem to get overridden or never forwarded. ``` val config = new Configuration() config.setString("fs.azure.account.key.mystore1.blob.core.windows.net<https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Ffs.azure.account.key.mystore1.blob.core.windows.net%2F&data=05%7C01%7CIvan.Webber%40microsoft.com%7C302a8781f35f4b64cfaf08db1b767f30%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638133968331853429%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000%7C%7C%7C&sdata=DWFU%2F86blheDCyBsWbtvZGbtk%2BcGrNmPoDNyvjIdkmM%3D&reserved=0>", System.getenv("KEY_1")) config.setString("fs.azure.account.key.mystore2.blob.core.windows.net<https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Ffs.azure.account.key.mystore2.blob.core.windows.net%2F&data=05%7C01%7CIvan.Webber%40microsoft.com%7C302a8781f35f4b64cfaf08db1b767f30%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638133968331853429%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000%7C%7C%7C&sdata=YiTsdw5gN4U5Y0jU6BBWadKQinf%2BtI3Wvu9EuKWn%2Bqc%3D&reserved=0>", System.getenv("KEY_2")) config.setString("fs.azure.account.key.mystore3.blob.core.windows.net<https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Ffs.azure.account.key.mystore3.blob.core.windows.net%2F&data=05%7C01%7CIvan.Webber%40microsoft.com%7C302a8781f35f4b64cfaf08db1b767f30%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638133968332009678%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000%7C%7C%7C&sdata=Z049XD4%2BJuSuk1BmSOD9umfket1hC90Oy6Q8jIQ9icY%3D&reserved=0>", System.getenv("KEY_3")) val env = environment.StreamExecutionEnvironment.getExecutionEnvironment(config) ``` In Flink operator configuration fields can be provided as follows, but then I can’t commit the file with a secret inside. Ideally there would be a way to reference a secret but the values must be literal strings. ``` spec: flinkConfiguration: fs.azure.account.key.mystore1.blob.core.windows.net<https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Ffs.azure.account.key.mystore1.blob.core.windows.net%2F&data=05%7C01%7CIvan.Webber%40microsoft.com%7C302a8781f35f4b64cfaf08db1b767f30%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638133968332009678%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000%7C%7C%7C&sdata=yALWxcdL1bNYNCnAUzSuhI9TEzYDGgxIdn9sW5OphfI%3D&reserved=0>: SECRET_STRING fs.azure.account.key.mystore2.blob.core.windows.net<https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Ffs.azure.account.key.mystore2.blob.core.windows.net%2F&data=05%7C01%7CIvan.Webber%40microsoft.com%7C302a8781f35f4b64cfaf08db1b767f30%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638133968332009678%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000%7C%7C%7C&sdata=ldqNKK1%2FtgDcPo6izcR6Jn53YqUMhGEAEs79%2BN8nf3A%3D&reserved=0>: SECRET_STRING fs.azure.account.key.mystore3.blob.core.windows.net<https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Ffs.azure.account.key.mystore3.blob.core.windows.net%2F&data=05%7C01%7CIvan.Webber%40microsoft.com%7C302a8781f35f4b64cfaf08db1b767f30%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638133968332009678%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000%7C%7C%7C&sdata=Z049XD4%2BJuSuk1BmSOD9umfket1hC90Oy6Q8jIQ9icY%3D&reserved=0>: SECRET_STRING ``` The last possible solution I can think that I’ll be trying is putting the entire flink-conf.yaml into a secret, or having a different container that adds secrets to the flink-operator-job.yaml and then does the `kubectl create -f flink-operator-job.yaml` (if that’s even possible).
