Re: dump Kafka to type 1 snapshot

2021-05-05 Thread Ryan Blue
Hi Lian, We've built most of this. I believe that it works to do this from Flink as long as you supply the columns that you want to use to identify rows for the upsert. We're also adding identity columns in table metadata so that you can do this through Flink SQL. That should work, but I should a

dump Kafka to type 1 snapshot

2021-05-04 Thread Lian Jiang
Hi, I want to dump events in a kafka topic into datalake as a type 1 snapshot in iceberg. Type 1 means a record having a key will overwrite the previous record having the same key. Each key will have only one record in the snapshot. Note that I want to simplify the long path: kafka -> (streaming