Sorry For the incomplete mail, sent bymistake I am working towards a incremental solution on hive based on the transient_lastDdlTime of the partitions. We mostly deal with hive external tables.
The transient_lastDdlTime of a partition gets updated when the insertion to the table happens via the insert query route, we are good there. But the issue is, if the file level updation happens in the partition folder, then hive doesnt update transient_lastDdlTime for that partition and we are not able to get the changed partitions list because of this. Unfortunately we cant change the way the hive table is being updated, its based on the file based update to the underlying location. When we do a file based ingestion, then we have the complete list of partitions updated. But this cannot be passed to the incremental system, hence our source of truth is hive metastore's a and its transient_lastDdlTime. Is there a way where I can update the transient_lastDdlTime in the metastore , for the partitions changed by adding files? I have tried to re-add the changed partition to the table, for updated ones so that the transient_lastDdlTime will change, but its not possible as it throws an already exists exception. Is there any other way? Thanks in advance. Thanks, Ravi On Tue, Aug 25, 2015 at 9:02 PM, ravi teja <raviort...@gmail.com> wrote: > Hi, > > I am working towards a incremental solution on hive based on the > transient_lastDdlTime of the partitions. > If the we in > > Thanks, > Ravi >
