Re: Migrating Sink v1 to v2

2022-09-28 Thread Yun Gao
Hi Krzysztof, I'll continue tracking the issue and hopefully it could be fixed before the next minor releases for 1.15.x, 1.16.x and 1.17. Best, Yun Gao -- From:Krzysztof Chmielewski Send Time:2022 Sep. 28 (Wed.) 23:46 To:dev ; Yun G

Re: Migrating Sink v1 to v2

2022-09-28 Thread Krzysztof Chmielewski
Yun Gao, Thank you very much for this. Do you see this ticked be picked up any time soon? For a moment I was thinking about trying to take my change with it but I guessing it would be rather hard without deeper understanding of Flink's internals, plus its marked as "major" Regards, Krzysztof Chmi

Re: Migrating Sink v1 to v2

2022-09-28 Thread Yun Gao
Hi Krzysztof, Very sorry for the long delay for it takes a bit of time to have a full investigation, the issue should be caused by the implementation bugs and I have filed an issue for the bugs. For the following actions, let's move to the thread of [2] for global synchronization. Very sorry for

Re: Migrating Sink v1 to v2

2022-09-09 Thread Krzysztof Chmielewski
It seems that there might be an issue with state recovery or some kind concurrency issue for GlobalCommitterOperator created by SinkV1Adapter that uses StandardSinkTopologies.addGlobalCommitter method in scenarios with cluster failover. I've tried to locate the place where the issue is and from

Re: Migrating Sink v1 to v2

2022-09-07 Thread Krzysztof Chmielewski
A small update, When I change number of Sinks from 3 to 1, test passes. śr., 7 wrz 2022 o 12:18 Krzysztof Chmielewski < krzysiek.chmielew...@gmail.com> napisał(a): > Hi, > I'm a co-author for opensource Delta-Flink connector hosted on [1]. > The connector was originated for Flink 1.12 and current