Re: Copying all Hive tables from Prod to UAT

That is a good point Jorn with regard to JDBC and Hive data I believe you can use JDBC to get a compressed data from an Oraclle or Sybase database cause decompression happens at the time of data access much like using a sqlplus or isql tool. However, it is worth trying what happens when one acces

Re: Copying all Hive tables from Prod to UAT

Hello, I've been looking at this recently for moving Hive tables from on-premise clusters to the cloud, but the principle should be the same for your use-case. If you wish to do this in an automated way, some tools worth considering are: - Hive's built in replication framework: https://cwik

Re: Copying all Hive tables from Prod to UAT

Or use Falcon ... The Spark JDBC I would try to avoid. Jdbc is not designed for these big data bulk operations, eg data has to be transferred uncompressed and there is the serialization/deserialization issue query result -> protocol -> Java objects -> writing to specific storage format etc This

Re: Copying all Hive tables from Prod to UAT

> We are using HDP. Is there any feature in ambari Apache Falcon handles data lifecycle management, not Ambari. https://falcon.apache.org/0.8/HiveDR.html Cheers, Gopal

Re: Copying all Hive tables from Prod to UAT

Thanks Mich. I will look into it. On 5/25/2016 9:05 AM, Mich Talebzadeh wrote: They are multiple ways of doing this without relying any vendors release. 1) Using hive EXPORT/IMPORT utility EXPORT TABLE table_or_partition TO hdfs_path; IMPORT [[EXTERNAL] TABLE table_or_partition] FROM

Re: Copying all Hive tables from Prod to UAT

We are using HDP. Is there any feature in ambari On 5/25/2016 6:50 AM, Suresh Kumar Sethuramaswamy wrote: Hi If you are using CDH, via CM , Backup->replications you could do inter cluster hive data transfer including metadata Regards Suresh On Wednesday, May 25, 2016, mahender bigdata m

Re: Copying all Hive tables from Prod to UAT

They are multiple ways of doing this without relying any vendors release. 1) Using hive EXPORT/IMPORT utility EXPORT TABLE table_or_partition TO hdfs_path; IMPORT [[EXTERNAL] TABLE table_or_partition] FROM hdfs_path [LOCATION [table_location]]; 2) This works for individual tables but you can ea

Re: Copying all Hive tables from Prod to UAT

Hi If you are using CDH, via CM , Backup->replications you could do inter cluster hive data transfer including metadata Regards Suresh On Wednesday, May 25, 2016, mahender bigdata wrote: > Any Document on it. > > On 4/8/2016 6:28 PM, Will Du wrote: > > did you try export and import statement

Re: Copying all Hive tables from Prod to UAT

Any Document on it. On 4/8/2016 6:28 PM, Will Du wrote: did you try export and import statement in HQL? On Apr 8, 2016, at 6:24 PM, Ashok Kumar > wrote: Hi, Anyone has suggestions how to create and copy Hive and Spark tables from Production to UAT. One way wo

Re: Copying all Hive tables from Prod to UAT

did you try export and import statement in HQL? > On Apr 8, 2016, at 6:24 PM, Ashok Kumar wrote: > > Hi, > > Anyone has suggestions how to create and copy Hive and Spark tables from > Production to UAT. > > One way would be to copy table data to external files and then move the > external fi