Re: Hive Concurrency support

Are you using a lock manager, and if so which one? I believe the ZooKeeper lock manager does not allow simultaneous writes. The lock manager that comes with the DbTxnManager does, but you can't use that without also using transactions. Alan. Suyog Parlikar

Re: Hive Concurrency support

No table is not transactional On Aug 24, 2015 10:30 AM, "Elliot West" wrote: > Is the table configured to be transactional? > > > https://cwiki.apache.org/confluence/display/Hive/Hive+Transactions#HiveTransactions-TableProperties > > On Monday, 24 August 2015, Suyog Parlikar wrote: > >> Hello El

Re: Hive Concurrency support

Is the table configured to be transactional? https://cwiki.apache.org/confluence/display/Hive/Hive+Transactions#HiveTransactions-TableProperties On Monday, 24 August 2015, Suyog Parlikar wrote: > Hello Elliot, > > Thanks for clarification. > > But I am still not able to understand the hive work

Re: Hive Concurrency support

Hello Elliot, Thanks for clarification. But I am still not able to understand the hive working. My cluster has following properties. Txn.manager- dummyTxnManager Cocurrency - true Actually I am trying to insert data into two different partitions of a table at the same time. When I check the

Re: Hive Concurrency support

correction in below: 2) You will have to coordinate concurrency via zookeeper for distributed >transactions. Without zookeeper or equivalent product it will not work >and you will end up with deadlocks in your metastore. Should read .. it will not work and you will end up with serialisation iss

Re: Hive Concurrency support

Well I have across this in practice with real time data movements DML inserts) using replication server to deliver data from RDBMS to Hive. In general if you have not met the conditions you will end up with deadlocks. To make this work you will need: 1) your Hive metastore must allow concurrency

Hive Concurrency support

Shared locks are all that is required to insert data into transactional tables. Multiple clients can hold a shared lock simultaneously. Each client will write using uniquely assigned transaction ids so that their work is isolated from one another. It should actually be possible for multiple clients

Re: Hive Concurrency support

If you are looking to support concurrency check this param: https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-hive.support.concurrency I believe it will allow to you run several different inserts to the same partitions, but I don't know what kind of

Re: Hive Concurrency support

Thanks Elliot, For the immediate reply. But as per hive locking mechanism, While inserting data to a partition hive acquires exclusive lock on that partition and shared lock on the entire table. How is it possible to insert data into a different partition of the same table while having shared lo

Re: Hive Concurrency support

I presume you mean "into different partitions of a table at the same time"? This should be possible. It is certainly supported by the streaming API, which is probably where you want to look if you need to insert large volumes of data to multiple partitions concurrently. I can't see why it would not

Hive Concurrency support

Can we insert data in different partitions of a table at a time. Waiting for inputs . Thanks in advance. - suyog