Re: CDC with "copy on write"

2020-12-22 Thread Ryan Blue
Hi Ashish, I think it makes sense for your use case (#1949) to expose a way to read overwrites using incremental scans, but I'm not sure how best to expose it. This is safe for your case because you know which records were deleted based on the ID, so you're basically replaying the data as incremen

Re: What's the time to expose iceberg format v2 to end users ?

2020-12-22 Thread Ryan Blue
Thanks, Yan! To summarize that doc a bit, the main blockers are: * Finish updating the spec for NaN counters and behavior * Fix the issue with partition transforms and values before 1970 (#1680) * Partition evolution: Add lastPartitionFieldId to table metadata and update docs * Add order id column