Hi Daniel, > The changelog type needs to be insert-only to invoke the correct streaming planner rules.
I mean that we can read all kind of data from mysql(insert, upsert, delete) as +I[optype, xxx] in flink (such as +I[insert, 1, 'a'], +I[delete, 2, null]). If downstream only needs +[insert, xx], a filter of optype is enough. Besides that, users can also do other things: 1. count the total number of each op_type 2. Do logical delete action in sink with flink sql( update value of is_delete in sink table if op_type is delete) Best, Hongshun On Fri, Aug 23, 2024 at 2:45 AM Daniel Henneberger <m...@danielhenneberger.com> wrote: > Unfortunately no, this would not be helpful. The changelog type needs to be > insert-only to invoke the correct streaming planner rules. > > On Wed, Aug 21, 2024 at 9:01 PM Hongshun Wang <loserwang1...@gmail.com> > wrote: > > > Hi Daniel， > > > > > > There's no way to convert a different changelog stream to an > insertonly > > > stream so I'm pushing this upstream to the connector. > > > > I wonder whether you just want the insert type changelog and ignore other > > kinds of changelog, or just want to read all kinds of changelog as insert > > in flink row data? If you want later, we need an `op_type` to distinguish > > the type of changelog, and users can filter depending on their need. > > > > Best, > > Hongshun > > [1] https://issues.apache.org/jira/browse/FLINK-35067 > > > > > > > > > > On Thu, Aug 22, 2024 at 9:29 AM Daniel Henneberger < > > m...@danielhenneberger.com> > > wrote: > > > > > Hey, > > > > > > I'd like to consume the postgres-cdc connector with an insert-only > > > changelog. I'm uninterested in any retraction messages since our use > case > > > doesn't require it, and we can greatly benefit from the flink > operations > > > that insert-only changelog provides. E.g. I want to do a tumble window > > > aggregation on the stream of events. We're using the table api > > exclusively > > > and there's no way to convert a different changelog stream to an insert > > > only stream so I'm pushing this upstream to the connector. I threw up a > > PR > > > to showcase the implementation. Let me know what you think of this. > > > > > > Thanks, > > > Daniel Henneberger > > > > > > https://github.com/apache/flink-cdc/pull/3562 > > > > > >
