Re: Discuss proposal - IRC APIs for Multi-Statement Multi-Table Transactions

Forgot to attach a link to the update proposal . On Tue, May 27, 2025 at 1:06 AM Maninderjit Singh < parmar.maninder...@gmail.com> wrote: > Hi community, > > I have updated

Re: Discuss proposal - IRC APIs for Multi-Statement Multi-Table Transactions

Hi community, I have updated the proposal with both the options (overwriting existing timestamps-ms vs introducing a new sequence/timestamp field) as we have initial consensus on using catalog authored sequence/timestamp. Jagdeep, please review to ensure that the options are correctly captured. I

Re: Discuss proposal - IRC APIs for Multi-Statement Multi-Table Transactions

I'm interested, but can't be there, but please record the meeting. Thanks, Peter Maninderjit Singh ezt írta (időpont: 2025. máj. 24., Szo, 2:30): > Hi dev community, > I was wondering if we could join a call next week for discussing the > multi-table transactions so we can make progress. I have

Re: Discuss proposal - IRC APIs for Multi-Statement Multi-Table Transactions

Hi dev community, I was wondering if we could join a call next week for discussing the multi-table transactions so we can make progress. I have shared a meeting invite where anyone who's interested in the discussion can join. Please let me know if this works. Thanks, Maninder Sync for iceberg mul

Re: Discuss proposal - IRC APIs for Multi-Statement Multi-Table Transactions

Hi dev community, Following up on the thread here to continue the discussion and get feedback since we couldn't get to it in sync. I think we have made some progress in the discussion that I want to capture while highlighting the items where we need to create consensus along with pros and cons. I w

Re: Discuss proposal - IRC APIs for Multi-Statement Multi-Table Transactions

Appreciate the feedback on the "catalog-authored timestamp" document ! Ryan, I don't think we can get consistent time travel queries in iceberg without fixing the timestamp field since it's what th

Re: Discuss proposal - IRC APIs for Multi-Statement Multi-Table Transactions

Thank you Ryan, Maninder and the rest of the community for feedback and ideas! Drew and I will take another pass and remove the catalog co-ordination requirement for LoadTable API, and bring the proposal closer to "catalog-authored timestamp" in the sense that clients can use CSN to find the right

Re: Discuss proposal - IRC APIs for Multi-Statement Multi-Table Transactions

Thanks for the proposals! There are things that I think are good about both of them. I think that the catalog-authored timestamps proposal misunderstands the purpose of the timestamp field, but does get right that a monotonically increasing "time" field (really a sequence number) across tables enab

Re: Discuss proposal - IRC APIs for Multi-Statement Multi-Table Transactions

Hi everyone, To avoid passing copies of a file around for comments, I put the doc for commit sequence numbers into Google so we can comment on a central copy: https://docs.google.com/document/d/1jr4Ah8oceOmo6fwxG_0II4vKDUHUKScb/edit?usp=sharing&ouid=100239850723655533404&rtpof=true&sd=true Ryan

Re: Discuss proposal - IRC APIs for Multi-Statement Multi-Table Transactions

Thanks for the updated proposal Drew! My preference for using the catalog authored timestamp is to minimize changes to the REST spec so we can have good backwards compatibility. I have quickly put together a draft proposal on how this should work. Looking forward to feedback and discussion. Draft

Re: Discuss proposal - IRC APIs for Multi-Statement Multi-Table Transactions

Hi everyone, Thank you for feedback on the MTT proposal and during community sync. Based on it, Jagdeep and I have iterated on the document and added a second option to use *Catalog CommitSequenceNumbers*. Looking forward to getting more feedback on the proposal, where to add more details or appro

Re: Discuss proposal - IRC APIs for Multi-Statement Multi-Table Transactions

Thanks for getting back to me and probing further! I agree that the approaches are functionally equivalent and we are debating using a logical clock which is time based vs non-time based. There are few good arguments that you have brought forward which I want to discuss further: *1. Catalog is most

Re: Discuss proposal - IRC APIs for Multi-Statement Multi-Table Transactions

Hi Maninder, I believe the two approaches are functionally equivalent, "Global CatalogSequenceNumbers" or "Catalog timestamps" that you propose. The difference is simply whether we use an actual timestamp or some monotonically increasing sequence/logical clock. I think we should use sequence numbe

Re: Discuss proposal - IRC APIs for Multi-Statement Multi-Table Transactions

Thanks for starting the discussion! I wanted to discuss more about the "Catalog GlobalSequenceNumber". I think we not only need to order the snapshots across tables but also assign them some timestamp so that time travel queries also work consistently across tables. In other words, "Catalog Global

Re: Discuss proposal - IRC APIs for Multi-Statement Multi-Table Transactions

Hi everyone, I enjoyed the discussion/feedback during Iceberg community sync. I am going through feedback and updating my proposal - adding the "Catalog GlobalSequenceNumber" option, and describing how it would look, detailing the use cases better in the document, and rest of feedback. Please also

Re: Discuss proposal - IRC APIs for Multi-Statement Multi-Table Transactions

Hi Amogh/Drew/Fokko and Iceberg community, Thank you for the feedback. As you point out, I will update the document and add a section to clarify what we mean by each isolation level. I read through the linked PR as well, this proposal is along the lines of *"1. Coordinate table loading with the ca

Re: Discuss proposal - IRC APIs for Multi-Statement Multi-Table Transactions

Hi Jagdeep, Thanks for the proposal! I agree with Drew’s second point emphasis on clarifying the goals for snapshot and serializable isolation guarantees before committing to catalog changes. Note a lot of the following rationale is also from this Github PR

Re: Discuss proposal - IRC APIs for Multi-Statement Multi-Table Transactions

Hey everyone, Thanks for putting this proposal together, Jagdeep! I just wanted to share a few quick thoughts from my side. There’ve been some past discussions around this topic, and we have some catalog implementations take on server-side responsibility for transaction logic for their own means.

Re: Discuss proposal - IRC APIs for Multi-Statement Multi-Table Transactions

Hi Fokko, Thank you for detailed feedback! I need to think through your comments and update/reply accordingly. But I want to discuss a key goal for writing this proposal (MTMS = "Multi-Table Multi-Statement"): For MTMS txns, many un-coordinated engines could be operating on the same set of tables

Re: Discuss proposal - IRC APIs for Multi-Statement Multi-Table Transactions

Hey Jagdeep, Thanks for proposing this. In general, it depends on what entity of the triangle (Table-format, Catalog, Engine) we want to make responsible for what operation. Where this proposal delegates more to the catalog, and what is done today in the engine itself. I went over it and added som