spark-submit fails after setting userClassPathFirst to true

2016-10-29 Thread spats
After i set spark.driver.userClassPathFirst=true, my spark-submit --master yarn-client fails with below error & it works fine if i remove userClassPathFirst setting. I need to add this setting to avoid class conflicts in some other job so trying to make it this setting work in simple job first & la

Issues with Spark On Hbase Connector and versions

2016-08-27 Thread spats
Regarding hbase connector by hortonworks https://github.com/hortonworks-spark/shc, it would be great if someone can answer these 1. What versions of Hbase & Spark expected? I could not run examples provided using spark 1.6.0 & hbase 1.2.0 2. I get error when i run example provided here

JavaRDD to DataFrame fails with null pointer exception in 1.6.0

2016-08-16 Thread spats
Cannot convert JavaRDD to DataFrame in spark 1.6.0, throws null pointer exception & no more details. Can't really figure out what really happening. Any pointer to fixes? //convert JavaRDD to DataFrame DataFrame schemaPeople = sqlContext.createDataFrame(people, Person.class); // exception with no