Delta Logic in Spark

Hi, We have daily data pull which pulls almost 50 GB of data from upstream system. We are using Spark SQL for processing of 50 GB. Finally insert 50 GB of data into Hive Target table and Now we are copying whole hive target table to SQL esp. SQL Staging Table & implement merge from staging

Re: Are Hive locks really ephemeral ?

Hi Rajesh, When the queries are locking tables/partitions they are using the acquireLocks method. If you are using DummyTxnManager then it acquires locks with keepAlive false. When using manual locking then that is a different story, and most probably you are right when you say that it creates