Vivek,
You might want to try MERGE INTO again. You should be able to make it more
efficient by adding predicates to the ON clause. Those will get pushed down
to the target table to avoid a big scan.
Iceberg supports transactions to do what you want, but it doesn't use table
locking. Instead, it w
Hey Ryan,
I was using spark merge into command but its performance became slower as
the size of data in the iceberg table was growing.
A part of the reason may be due to I need to overwrite some of the older
partitions often as a side effect of delete and update.
Another reason may be that *MERG
On 2021/05/02 18:10:19, Ryan Blue wrote:
> Vivek,
>
> Currently, Spark doesn't support any of the BEGIN/COMMIT statements for
> transactions, so I don't think that it is possible right now. What are you
> trying to do? It may be that some of the newer commands, like MERGE INTO,
> would work f
Vivek,
Currently, Spark doesn't support any of the BEGIN/COMMIT statements for
transactions, so I don't think that it is possible right now. What are you
trying to do? It may be that some of the newer commands, like MERGE INTO,
would work for you instead.
On Thu, Apr 29, 2021 at 5:49 PM vivek B
Hey All,
Is there a way to run multiple sql operations via spark as one single
transaction ?
Thanks,
vivek
