Hello Apache Hive Community,
Hope you are doing well.
My use-case involves having a central Hive 4 HMS on a RDS (mysql) instance.
I also have some Hive 3 clusters (whose schema I cannot upgrade to Hive 4
yet). I am running queries from HS2 on the Hive 3 clusters which results in
HMS calls to the central Hive 4 metastore.
I wanted to understand if HMS 4 is backwards compatible with Hive 3.
Additionally I have found the following incompatible changes so far (the
list is not exhaustive) ->
- Field 'ENGINE' doesn't have a default value in TAB_COL_STATS
- Fixed by configuring it with a default value
- ALTER TABLE TAB_COL_STATS MODIFY COLUMN ENGINE VARCHAR(128)
DEFAULT ‘hive’;
- Field 'ENGINE' doesn't have a default value) in PART_COL_STATS
- Fixed by configuring it with a default value
- ALTER TABLE PART_COL_STATS MODIFY COLUMN ENGINE VARCHAR(128)
DEFAULT ‘hive’;
- org.apache.hadoop.hive.metastore.api.MetaException: Unable to
select from transaction database java.sql.SQLSyntaxErrorException:
(conn=328) Table 'hive.NEXT_TXN_ID' doesn't exist
- This is because https://issues.apache.org/jira/browse/HIVE-23048
renamed certain core HMS tables related to ACID. For instance,
*NEXT_TXN_ID
*has been renamed to *TXN_LOCK_TBL*. The column *NTXN_NEXT* has been
renamed to* TXN_LOCK.* This change breaks the backward compatibility
between HMS 4 and HMS 3 schemas. Do we have any context on why we renamed
the tables
Could you please let me know if Hive 4 HMS schema is backward compatible
with Hive 3. Is there any official doc where I can get the compatibility
info. How could we ensure that we have backwards compatibility between HMS
in Hive 4 and that of Hive 3.
Any guidance on this is greatly appreciated.
Thanks,
Anmol
