I’m attempting to write to Azure Blob Storage using Flink's FileOutputFormat.
I’ve included
hadoop-azure<https://hadoop.apache.org/docs/current/hadoop-azure/index.html#Configuring_Credentials>
within the jar I submit to Flink and configured the paths to be prefixed with
wasb://{CONTAINERNAME}@{ACCOUNTNAME}.blob.core.windows.net/<http://blob.core.windows.net/>.
When the file output format initializes, I get the following error: ERROR ROOT
- Run 4bfb099a-8d07-11e7-8d3a-fb4d07562cc0 failed with error:
'org.apache.flink.client.program.ProgramInvocationException: The program
execution failed: Cannot initialize task 'DataSink (/out/data)': No file system
found with scheme wasb, referenced in file URI
'wasb://blob@{ACCOUNTNAME}.blob.core.windows.net/<http://blob.core.windows.net/>out/data’.
Can I register the format programmatically from within the job (without putting
credentials into a core-site.xml file on the task manager)? Can I still use
Flink’s FileOutputFormat or should I be using a Hadoop OutputFormat?
Thanks,
Joshua
