[jira] [Created] (FLINK-24435) FsStateChangelogWriter#lastAppendedSequenceNumber return different seq number with no writes

2021-10-01 Thread Yuan Mei (Jira)
Yuan Mei created FLINK-24435: Summary: FsStateChangelogWriter#lastAppendedSequenceNumber return different seq number with no writes Key: FLINK-24435 URL: https://issues.apache.org/jira/browse/FLINK-24435

[jira] [Created] (FLINK-24436) FsStateChangelogWriter#lastAppendedSequenceNumber return different seq number with no writes

2021-10-01 Thread Yuan Mei (Jira)
Yuan Mei created FLINK-24436: Summary: FsStateChangelogWriter#lastAppendedSequenceNumber return different seq number with no writes Key: FLINK-24436 URL: https://issues.apache.org/jira/browse/FLINK-24436

Re: The Apache Flink should pay more attention to ensuring API compatibility.

2021-10-01 Thread Jark Wu
Hi all, Nice thread and great discussion! Ecosystem is one of the most important things to the Flink community, we should pay more attention to API compatibility. Marking all connector APIs @Public is a good idea, not only mark the Table/SQL connector APIs public, but also the new Source/Sink API

[jira] [Created] (FLINK-24437) Remove unhandled exception handler from CuratorFramework before closing it

2021-10-01 Thread Till Rohrmann (Jira)
Till Rohrmann created FLINK-24437: - Summary: Remove unhandled exception handler from CuratorFramework before closing it Key: FLINK-24437 URL: https://issues.apache.org/jira/browse/FLINK-24437 Project:

Re: The Apache Flink should pay more attention to ensuring API compatibility.

2021-10-01 Thread Ingo Bürk
Hi, > [...] but also the new Source/Sink APIs as public I'm not really involved in the new Source/Sink APIs and will happily listen to the developers working with them here, but since they are new, we should also be careful not to mark them as stable too quickly. We've only begun updating the exi

Re: [Flink blogs]

2021-10-01 Thread Etienne Chauchot
Hi all, Thanks a lot for your feedback guys ! Special thanks to Fabian, Till and Arvid (in a private discussion) ! The consensus seems to go toward the blog post on migrating a batch pipeline from DataSet API to DataStream API. For the record it is linked to a work I did lately (unfortunatel

[jira] [Created] (FLINK-24438) Port Kinesis Source to new Source API (FLIP-27)

2021-10-01 Thread Martijn Visser (Jira)
Martijn Visser created FLINK-24438: -- Summary: Port Kinesis Source to new Source API (FLIP-27) Key: FLINK-24438 URL: https://issues.apache.org/jira/browse/FLINK-24438 Project: Flink Issue Typ

Re: The Apache Flink should pay more attention to ensuring API compatibility.

2021-10-01 Thread Arvid Heise
The issue is that if we do not mark them as Public, we will always have incompatibilities. The change of SourceReaderContext#metricGroup is perfectly fine according to the annotation. The implications that we see here just mean that the interfaces have been expected to be Public. And now the quest

Re: The Apache Flink should pay more attention to ensuring API compatibility.

2021-10-01 Thread Jark Wu
Hi Arvid, > Should we expect connector devs to release different connector binaries for different Flink minors? >From the discussion of this thread, I think the answer is obviously "not", otherwise OpenInx won't start this discussion. As a maintainer of flink-cdc-connector, I have to say that it'

[jira] [Created] (FLINK-24440) Announce and combine latest watermarks across SourceReaders

2021-10-01 Thread Piotr Nowojski (Jira)
Piotr Nowojski created FLINK-24440: -- Summary: Announce and combine latest watermarks across SourceReaders Key: FLINK-24440 URL: https://issues.apache.org/jira/browse/FLINK-24440 Project: Flink

[jira] [Created] (FLINK-24441) Block SourceReader when watermarks are out of alignment

2021-10-01 Thread Piotr Nowojski (Jira)
Piotr Nowojski created FLINK-24441: -- Summary: Block SourceReader when watermarks are out of alignment Key: FLINK-24441 URL: https://issues.apache.org/jira/browse/FLINK-24441 Project: Flink I

[jira] [Created] (FLINK-24439) Introduce a CoordinatorStore

2021-10-01 Thread Piotr Nowojski (Jira)
Piotr Nowojski created FLINK-24439: -- Summary: Introduce a CoordinatorStore Key: FLINK-24439 URL: https://issues.apache.org/jira/browse/FLINK-24439 Project: Flink Issue Type: Sub-task

Re: The Apache Flink should pay more attention to ensuring API compatibility.

2021-10-01 Thread Piotr Nowojski
Hi, I don't understand why we are talking about this being a blocker issue? New sources were not marked as @Public for a good reason until 1.14. I agree, we should try better at making APIs @Public sooner. I was even proposing to create strict timeout rules (enforced via some automated checks) lik