Vihang Karajgaonkar created HIVE-21115:
------------------------------------------
Summary: Add support for object versions in metastore
Key: HIVE-21115
URL: https://issues.apache.org/jira/browse/HIVE-21115
Project: Hive
Issue Type: Improvement
Reporter: Vihang Karajgaonkar
Currently, metastore objects are identified uniquely by their names (eg.
catName, dbName and tblName for a table is unique). Once a table or partition
is created it could be altered in many ways. There is no good way currently to
identify the version of the object once it is altered. For example, suppose
there are two clients (Hive and Impala) using the same metastore. Once some
alter operations are performed by a client, another client which wants to do a
alter operation has no good way to know if the object which it has is the same
as the one stored in metastore. Metastore updates the {{transient_lastDdlTime}}
every time there is a DDL operation on the object. However, this value cannot
be relied for all the clients since after HIVE-1768 metastore updates the value
only when it is not set in the parameters. It is possible that a client which
alters the object state, does not remove the {{transient_lastDdlTime}} and
metastore will not update it. Secondly, if there is a clock skew between
multiple HMS instances when HMS-HA is configured, time values cannot be relied
on to find out the sequence of alter operations on a given object.
This JIRA propose to use JDO versioning support by Datanucleus
http://www.datanucleus.org/products/accessplatform_4_2/jdo/versioning.html to
generate a incrementing sequence number every time a object is altered. The
value of this object can be set as one of the values in the parameters. The
advantage of using Datanucleus the versioning can be done across HMS instances
as part of the database transaction and it should work for all the supported
databases.
In theory such a version can be used to detect if the client is presenting a
object which is "stale" when issuing a alter request. Metastore can choose to
reject such a alter request since the client may be caching a old version of
the object and any alter operation on such stale object can potentially
overwrite previous operations. However, this is can be done in a separate JIRA.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
