Thanks a lot Noam, you are a saviour! Ravi
On Tue, Aug 25, 2015 at 10:03 PM, Noam Hasson <noam.has...@kenshoo.com> wrote: > Hi, > > Check if this helps you: > > https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-AlterTable/PartitionTouch > > Noam. > > On Tue, Aug 25, 2015 at 6:43 PM, ravi teja <raviort...@gmail.com> wrote: > >> Sorry For the incomplete mail, sent bymistake >> >> I am working towards a incremental solution on hive based on the >> transient_lastDdlTime of the partitions. >> We mostly deal with hive external tables. >> >> The transient_lastDdlTime of a partition gets updated when the insertion >> to the table happens via the insert query route, we are good there. >> >> But the issue is, if the file level updation happens in the partition >> folder, then hive doesnt update transient_lastDdlTime for that partition >> and we are not able to get the changed partitions list because of this. >> >> >> Unfortunately we cant change the way the hive table is being updated, its >> based on the file based update to the underlying location. >> When we do a file based ingestion, then we have the complete list of >> partitions updated. >> But this cannot be passed to the incremental system, hence our source of >> truth is hive metastore's a and its transient_lastDdlTime. >> >> Is there a way where I can update the transient_lastDdlTime in the >> metastore , for the partitions changed by adding files? >> I have tried to re-add the changed partition to the table, for updated >> ones so that the transient_lastDdlTime will change, but its not possible >> as it throws an already exists exception. >> >> Is there any other way? >> Thanks in advance. >> >> Thanks, >> Ravi >> >> On Tue, Aug 25, 2015 at 9:02 PM, ravi teja <raviort...@gmail.com> wrote: >> >>> Hi, >>> >>> I am working towards a incremental solution on hive based on the >>> transient_lastDdlTime of the partitions. >>> If the we in >>> >>> Thanks, >>> Ravi >>> >> >> > > This e-mail, as well as any attached document, may contain material which > is confidential and privileged and may include trademark, copyright and > other intellectual property rights that are proprietary to Kenshoo Ltd, > its subsidiaries or affiliates ("Kenshoo"). This e-mail and its > attachments may be read, copied and used only by the addressee for the > purpose(s) for which it was disclosed herein. If you have received it in > error, please destroy the message and any attachment, and contact us > immediately. If you are not the intended recipient, be aware that any > review, reliance, disclosure, copying, distribution or use of the contents > of this message without Kenshoo's express permission is strictly prohibited.
