Hi Anshuman,
You can have a cron job that calls the (MSCK REPAIR ...table) script for the
table in Shellscript/Python (whatever your prefer).
Apache Nifi has a scheduling option as well. Maybe you can use it?
ThanksSrabasti Banerjee
On Friday, 24 March 2017 7:40 AM, Anshuman Ghosh
<anshuman.ghosh2...@gmail.com> wrote:
Hello Team,
This is Anshu here, nice to meet you!
I am working at Crealytics, Berlin as a Big Data Consultant/ Solution
Architect.
We are building/ designing a data flow pipeline using mainly *Apache NiFi
--> Consume data from Kafka --> Transform, Convert and Merge --> Write to
HDFS*
We have several *HIVE External table*s defined to point to HDFS location
where we are dumping the data.
All these tables are partitioned (mostly by Date); through NiFi we are able
to create and write to a new partition.
However we need to execute a meta refresh (MSCK REPAIR ..) in order for
HIVE to pick the new partition.
Is there any way to automate this process?
We can have a small script to run and do the same but then we wouldn't know
the run frequency of it (in case of a partition apart from Date).
Thanking you in advance!
______________________
*Kind Regards,*
*Anshuman Ghosh*
*Contact - +49 179 9090964*
