Thanks for the pointers. Yes, i started with changing the hive.group property in pom and started seeing various dependency issues.
Initially i thought spark-project.hive was just a pom for uber jars that pull in hive classes without transitive dependencies like kryo, but looks like lot more changes are needed, including editing the sources. We are looking at alternative approaches, since our customizations to Hive are pretty limited and may not warrant the effort required here. Thanks. On Tue, Aug 11, 2015 at 2:29 PM, Steve Loughran <ste...@hortonworks.com> wrote: > > On 11 Aug 2015, at 12:25, Pala M Muthaia <mchett...@rocketfuelinc.com> > wrote: > > Hi, > > I am trying to make Spark SQL 1.4 work with our internal fork of Hive. We > have some customizations in Hive (custom authorization, various hooks etc) > that are all part of hive-exec. > > Given Spark's hive dependency is through org.spark-project.hive groupId, > looks like i need to modify the definition of hive-exec artifact there to > take dependency on our internal hive (vs org.apache.hive), and then > everything else would flow through. > > > you can just change the hive group definition in your spark build; that's > the easy part > <hive.group>org.spark-project.hive</hive.group> > > harder is getting a consistent kryo binding and any other > shading/unshading. In SPARK-8064 we've moved Spark 1.5 to using Hive 1.2.1, > but even there we had to patch hive to use the same Kryo version, and shade > protobuf in hive-exec for everything to work on Hadoop 1.x. > > > However, i am unable to find sources for org.spark-project.hive to make > this change. Is it available? Otherwise, how can i proceed in this > situation? > > > Ted's pointed to the 0.13 code; the 1.2.1 is under > https://github.com/pwendell/hive/commits/release-1.2.1-spark > > however: do not attempt to change hive versions in a release, things are > intertwined at the SparkSQL level your code just won't work. > > > > Thanks, > pala > > >
