Re: Read schema to use during time travel

2021-11-22 Thread Ajantha Bhat
Hi Jack, Thanks for bringing up this discussion. but another schema could be used if the user specifies the schema ID. I think it is hard for users to specify the schema id. I think we don't even have a metadata table to see how schema is evolved per operation. So, Unless user knows which table m

Re: Read schema to use during time travel

2021-11-22 Thread Jack Ye
Thanks for the reply Ryan, and thanks for the additional information Mayur! Yes, the example you described is one of the biggest reasons for using the snapshot schema. I just brought the other example up because that was the conflicting use case I heard. The use case is that the system would alway

RE: Read schema to use during time travel

2021-11-22 Thread Mayur Srivastava
issues (as Ryan mentioned earlier) and works only when the current schema (or a future schema) is backwards compatible with snapshot schema. Thanks, Mayur From: Ryan Blue Sent: Monday, November 22, 2021 6:41 PM To: Iceberg Dev List Subject: Re: Read schema to use during time travel Thanks, Jack

Re: Read schema to use during time travel

2021-11-22 Thread Ryan Blue
Thanks, Jack! I was just thinking about this because I think we want to have support for time travel with the snapshot schema in 0.13.0 if possible, so it's a great time to talk about the behavior. In general, I think that time travel with the snapshot schema is the right option. The use case I th

Read schema to use during time travel

2021-11-22 Thread Jack Ye
Hi everyone, As we are starting to have more use cases around the time travel related features in Iceberg, a few questions popped up around the correct schema to use for time travel queries. *Latest table schema vs schema at snapshot time* There was a discussion around this in https://github.com/