Hi, This is as a matter of information.
I have seen some threads in stackoverflow about issues accessing Hive remotely without using the locality (Spark and Hive on the same Haddop cluster) or using hive-site.xml under $SPARK/conf. That process works fine. However, challenges come when accessing Hive on-premise from Cloud through PySpark etc. I worked on it a while back and today had to revisit it. The problem appears to be the classical JDBC drivers shipped with Hive treat Hive tables like a csv file, with header returned only. The signature is from df.printSchema() that displays <table_name>.<column_name> as opposed to column name only. I tried all sort of vendors drivers and the only one seem to work is Cloudera supplied driver for Hive <https://www.cloudera.com/downloads/connectors/hive/jdbc/2-6-5.html>.The driver is called com.cloudera.hive.jdbc41.HS2Driver and you need Hive connection URL in my case jdbc:hive2://<HOST_NAME>:10099/default The jar file is called HiveJDBC41.jar and all you need to do is to put it under $SPARK_HOME/jars, nothing else. To read the table jdbcHive = spark.read. \ format("jdbc"). \ option("url", config['hiveVariables']['hive_url']). \ option("dbtable", fullyQualifiedTableName). \ option("user", config['hiveVariables']['hive_user']). \ option("password", config['hiveVariables']['hive_password']). \ option("driver", config['hiveVariables']['hive_driver']). \ option("fetchsize", "1000"). \ load() fullyQualifiedTableName composed of <hive_database>.<hive_table>. I have provided a generous fetchsize. anything >= 20 should do. HTH LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* *Disclaimer:* Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction.
