On 11 Aug 2015, at 12:25, Pala M Muthaia <mchett...@rocketfuelinc.com<mailto:mchett...@rocketfuelinc.com>> wrote:
Hi, I am trying to make Spark SQL 1.4 work with our internal fork of Hive. We have some customizations in Hive (custom authorization, various hooks etc) that are all part of hive-exec. Given Spark's hive dependency is through org.spark-project.hive groupId, looks like i need to modify the definition of hive-exec artifact there to take dependency on our internal hive (vs org.apache.hive), and then everything else would flow through. you can just change the hive group definition in your spark build; that's the easy part <hive.group>org.spark-project.hive</hive.group> harder is getting a consistent kryo binding and any other shading/unshading. In SPARK-8064 we've moved Spark 1.5 to using Hive 1.2.1, but even there we had to patch hive to use the same Kryo version, and shade protobuf in hive-exec for everything to work on Hadoop 1.x. However, i am unable to find sources for org.spark-project.hive to make this change. Is it available? Otherwise, how can i proceed in this situation? Ted's pointed to the 0.13 code; the 1.2.1 is under https://github.com/pwendell/hive/commits/release-1.2.1-spark however: do not attempt to change hive versions in a release, things are intertwined at the SparkSQL level your code just won't work. Thanks, pala
