Hi Ivan, Mercy is always free. Are you using WASB or ABFS? I presume it's the latter, since that's the one that can't use EnvironmentVariableKeyProvider, but just to be sure.
Regards, Alexis. On Thu, 2 Mar 2023, 23:07 Ivan Webber via user, <user@flink.apache.org> wrote: > TLDR: I will buy your coffee if you can help me understand to securely > configure Azure credentials (doc page > <https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/filesystems/azure/> > for reference). > > > > I am a junior developer tasked with being the first person to learn the > Apache Flink framework. I know that storing secrets in flink-conf.yaml in a > container is a bad idea. I’ve tried exposing Azure storage keys as env vars > and using `config.setString`, but those properties seem to get overridden. > I plan on using Flink operator, so if you can show me in that context > that’d be ideal. > > > > Thanks, and sorry for bothering everyone. I’ve just exhausted myself and > am hopeful someone will have mercy for me. I really will Venmo you $5 for > coffee if you want. > > > > Thanks, > > > > Ivan > > > > > > Larger code examples: > > > > Setting dynamic properties before executing the job doesn’t work because > the values seem to get overridden or never forwarded. > > ``` > > val config = new Configuration() > > config.setString("fs.azure.account.key.mystore1.blob.core.windows.net", > System.getenv("KEY_1")) > > config.setString("fs.azure.account.key.mystore2.blob.core.windows.net", > System.getenv("KEY_2")) > > config.setString("fs.azure.account.key.mystore3.blob.core.windows.net", > System.getenv("KEY_3")) > > val env = environment.StreamExecutionEnvironment > .getExecutionEnvironment(config) > > ``` > > > > In Flink operator configuration fields can be provided as follows, but > then I can’t commit the file with a secret inside. Ideally there would be a > way to reference a secret but the values must be literal strings. > > ``` > > spec: > > flinkConfiguration: > > fs.azure.account.key.mystore1.blob.core.windows.net: SECRET_STRING > > fs.azure.account.key.mystore2.blob.core.windows.net: SECRET_STRING > > fs.azure.account.key.mystore3.blob.core.windows.net: SECRET_STRING > > ``` > > > > The last possible solution I can think that I’ll be trying is putting the > entire flink-conf.yaml into a secret, or having a different container that > adds secrets to the flink-operator-job.yaml and then does the `kubectl > create -f flink-operator-job.yaml` (if that’s even possible). >
