Hi,
I'm running Spark 1.4 on Azure.
DataFrame's insertInto fails, but when saveAsTable works.
It seems like some issue with accessing Azure's blob storage but that
doesn't explain why one type of write works and the other doesn't.
This is the stack trace:
Caused by: org.apache.hadoop.fs.azure.AzureException:
org.apache.hadoop.fs.azure.KeyProviderException: Unable to load key
provider class.
at
org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.createAzureStorageSession(AzureNativeFileSystemStore.java:938)
at
org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.initialize(AzureNativeFileSystemStore.java:438)
at
org.apache.hadoop.fs.azure.NativeAzureFileSystem.initialize(NativeAzureFileSystem.java:1048)
at
org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2596)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:91)
at
org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2630)
at
org.apache.hadoop.fs.FileSystem$Cache.getUnique(FileSystem.java:2618)
at org.apache.hadoop.fs.FileSystem.newInstance(FileSystem.java:417)
at
org.apache.hadoop.hive.shims.Hadoop23Shims.getNonCachedFileSystem(Hadoop23Shims.java:574)
at
org.apache.hadoop.hive.ql.exec.Utilities.createDirsWithPermission(Utilities.java:3424)
at
org.apache.hadoop.hive.ql.exec.Utilities.createDirsWithPermission(Utilities.java:3396)
at org.apache.hadoop.hive.ql.Context.getScratchDir(Context.java:214)
... 59 more
Caused by: org.apache.hadoop.fs.azure.KeyProviderException: Unable to load
key provider class.
at
org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.getAccountKeyFromConfiguration(AzureNativeFileSystemStore.java:829)
at
org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.createAzureStorageSession(AzureNativeFileSystemStore.java:917)
... 70 more
Caused by: java.lang.ClassNotFoundException: Class
org.apache.hadoop.fs.azure.ShellDecryptionKeyProvider not found
at
org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1980)
at
org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.getAccountKeyFromConfiguration(AzureNativeFileSystemStore.java:826)
... 71 more
Thanks,
Daniel
