Re: Concurrent transactions

2022-02-19 Thread Thippana Vamsi Kalyan
Thanks Russell and Ryan for clarification on Transaction usage. What do you think about passing a special table ops that does not refresh from catalog unless a commit was called on it? So, that way the base in Transaction never moves and during commit a conflict will be found, and then because my

Re: Concurrent transactions

2022-02-19 Thread Ryan Blue
The problem that you’re hitting is that the initial commit succeeds. It doesn’t have anything to do with retries. The commit is started when you call commitTransaction not when you create the transaction. That’s because transactions aren’t watching the table state from the transaction start. Inste

Re: Concurrent transactions

2022-02-19 Thread Thippana Vamsi Kalyan
> > you basically get 1 retry for free off the bat, > Yes, I am getting this free retry even though I am setting "commit.retry.num-retries" to zero 2) Creating a transaction that wraps a base transaction and that overrides > commitTransaction to just fail if the metadata from when the object was

Re: Concurrent transactions

2022-02-19 Thread Russell Spitzer
The issue with the above example is that starting a transaction is not a shared state amongst iceberg clients or the table itself. There is nothing that other clients could know about or check. If you look at the code for committing a transaction, it starts building the transaction off of whatever

Concurrent transactions

2022-02-19 Thread Thippana Vamsi Kalyan
Hi, I am trying to understand the usage of Transactions in Iceberg with "commit.retry.num-retries" set to zero. My requirement is that the transaction must fail if the table gets updated by any concurrent transaction after opening the transaction. I wrote the following unit test in TestHadoopTabl