Re: New Spark Datasource for Hive ACID tables

2019-07-26 Thread Abhishek Somani
Hey Naresh, Thanks for your question. Yes it will work! Thanks, Abhishek Somani On Fri, Jul 26, 2019 at 7:08 PM naresh Goud wrote: > Thanks Abhishek. > > Will it work on hive acid table which is not compacted ? i.e table having > base and delta files? > > Let’s say hive a

New Spark Datasource for Hive ACID tables

2019-07-26 Thread Abhishek Somani
e tables via Spark as well. The datasource is also available as a spark package, and instructions on how to use it are available on the Github page <https://github.com/qubole/spark-acid>. We welcome your feedback and suggestions. Thanks, Abhishek Somani

Re: Custom datasource: when acquire and release a lock?

2019-05-27 Thread Abhishek Somani
method? > > What technology is used in the transactional data endpoint? > > > > Am 24.05.2019 um 15:36 schrieb Abhishek Somani < > abhisheksoman...@gmail.com>: > > > > Hi experts, > > > > I am trying to create a custom Spark Datasource(v1) to read fr

Re: Custom datasource: when acquire and release a lock?

2019-05-26 Thread Abhishek Somani
Hi experts, I'll be very grateful if someone could help. Thanks, Abhishek On Fri, May 24, 2019 at 7:06 PM Abhishek Somani wrote: > Hi experts, > > I am trying to create a custom Spark Datasource(v1) to read from a > transactional data endpoint, and I need to acquire a lock

Custom datasource: when acquire and release a lock?

2019-05-24 Thread Abhishek Somani
called again as the RDD would be reused and the partitions would have gotten cached in the RDD. Can someone advice me on where would be the right places to acquire and release a lock with my data endpoint in this scenario. Thanks a lot, Abhishek Somani