Re: Flink CDC insert-only changelog mode

2024-08-25 Thread Hongshun Wang
Hi Daniel, > The changelog type needs to be insert-only to invoke the correct streaming planner rules. I mean that we can read all kind of data from mysql(insert, upsert, delete) as +I[optype, xxx] in flink (such as +I[insert, 1, 'a'], +I[delete, 2, null]). If downstream only needs +[insert, xx

Re: Flink CDC insert-only changelog mode

2024-08-22 Thread Daniel Henneberger
Unfortunately no, this would not be helpful. The changelog type needs to be insert-only to invoke the correct streaming planner rules. On Wed, Aug 21, 2024 at 9:01 PM Hongshun Wang wrote: > Hi Daniel, > > > > There's no way to convert a different changelog stream to an insertonly > > stream so

Re: Flink CDC insert-only changelog mode

2024-08-21 Thread Hongshun Wang
Hi Daniel, > > There's no way to convert a different changelog stream to an insertonly > stream so I'm pushing this upstream to the connector. I wonder whether you just want the insert type changelog and ignore other kinds of changelog, or just want to read all kinds of changelog as insert in fli

Flink CDC insert-only changelog mode

2024-08-21 Thread Daniel Henneberger
Hey, I'd like to consume the postgres-cdc connector with an insert-only changelog. I'm uninterested in any retraction messages since our use case doesn't require it, and we can greatly benefit from the flink operations that insert-only changelog provides. E.g. I want to do a tumble window aggregat