Hi Ravi, What exactly are you trying to do?
You want to enhance Spark SQl or you want to run Hive on Spark engine? HTH LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* *Disclaimer:* Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction. On Thu, 22 Oct 2020 at 16:36, Ravi Shankar <[email protected]> wrote: > Hello all, > I am trying to understand how the Spark SQL integration with hive works. > Whenever i build spark with -Phive -P hive-thriftserver options, i see that > it is packaged with hive-2.3.7*.jars and spark-hive*.jars. And the > documentation claims that spark can talk to different versions of hive. If > that is the case , what should i do if i have a hive 3.2.1 running on my > instance and i want my spark application to talk to that hive cluster. > > Does this mean i have to build spark with hive version 3.2.1 or like > the documentation states, is it enough if i just add the metastore jars to > spark-defaults.conf ? > > Should i add my hive 3.2.1 lib to the SPARK_DIST_CLASSPATH as well ? Will > there be conflicts between the hive 2.3.7 jars and the hive 3.2.1 jars i > will have in this case ? > > > Thanks ! >
