Handling record level TTL in CDC ingestion in Iceberg

Hi All, We are exploring CDC ingestion from DBs to Iceberg, certain DB like TiDB have row level TTL enabled, there seem to be following ways to handle ttl- 1. Emit an explicit CDC event for the row as soon as the TTL window expires 2. Handle TTL at reader layer by filtering expired rows in Iceberg

Re: Query regarding UPSERT Mode in Flink

s just come up in > the discussion about the table maintenance thread: > https://lists.apache.org/thread/37k03vfmrr81o10d9zgdhpyc28mrrp9z > > Aditya Narayan Gupta ezt írta (időpont: > 2024. ápr. 22., H, 15:28): > >> Hi Péter, >> >> Thanks for the detailed answers

Re: Query regarding UPSERT Mode in Flink

procedure run at higher frequency vs having run compaction? or are there any other suggestions? Regards, Aditya On Mon, Apr 22, 2024 at 2:35 PM Péter Váry wrote: > Hi Aditya, > > See my answers below: > > Aditya Narayan Gupta ezt írta (időpont: > 2024. ápr. 20., Szo, 11:05): &

Re: Query regarding UPSERT Mode in Flink

Hi Péter, Gabor, Thanks a lot for clarifying and providing additional information, I had few followup queries- 1. We want to ingest an CDC stream using Flink to Iceberg sink, if we have a RETRACT like CDC stream that uniquely distinguishes between INSERT / UPDATE (for eg contains events as define