Re: Spark 1.4 HiveContext fails to initialise with native libs

2015-06-04 Thread Yin Huai
Are you using RC4? On Wed, Jun 3, 2015 at 10:58 PM, Night Wolf wrote: > Thanks Yin, that seems to work with the Shell. But on a compiled > application with Spark-submit it still fails with the same exception. > > On Thu, Jun 4, 2015 at 2:46 PM, Yin Huai wrote: > >> Can you put the following set

Re: Spark 1.4 HiveContext fails to initialise with native libs

2015-06-03 Thread Night Wolf
Thanks Yin, that seems to work with the Shell. But on a compiled application with Spark-submit it still fails with the same exception. On Thu, Jun 4, 2015 at 2:46 PM, Yin Huai wrote: > Can you put the following setting in spark-defaults.conf and try again? > > spark.sql.hive.metastore.sharedPref

Re: Spark 1.4 HiveContext fails to initialise with native libs

2015-06-03 Thread Yin Huai
Can you put the following setting in spark-defaults.conf and try again? spark.sql.hive.metastore.sharedPrefixes com.mysql.jdbc,org.postgresql,com.microsoft.sqlserver,oracle.jdbc,com.mapr.fs.shim.LibraryLoader,com.mapr.security.JNISecurity,com.mapr.fs.jni https://issues.apache.org/jira/browse/SPAR

Spark 1.4 HiveContext fails to initialise with native libs

2015-06-03 Thread Night Wolf
Hi all, Trying out Spark 1.4 RC4 on MapR4/Hadoop 2.5.1 running in yarn-client mode with Hive support. *Build command;* ./make-distribution.sh --name mapr4.0.2_yarn_j6_2.10 --tgz -Pyarn -Pmapr4 -Phadoop-2.4 -Pmapr4 -Phive -Phadoop-provided -Dhadoop.version=2.5.1-mapr-1501 -Dyarn.version=2.5.1-mapr