[jira] [Created] (FLINK-24435) FsStateChangelogWriter#lastAppendedSequenceNumber return different seq number with no writes
Yuan Mei created FLINK-24435: Summary: FsStateChangelogWriter#lastAppendedSequenceNumber return different seq number with no writes Key: FLINK-24435 URL: https://issues.apache.org/jira/browse/FLINK-24435 Project: Flink Issue Type: Bug Reporter: Yuan Mei -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-24436) FsStateChangelogWriter#lastAppendedSequenceNumber return different seq number with no writes
Yuan Mei created FLINK-24436: Summary: FsStateChangelogWriter#lastAppendedSequenceNumber return different seq number with no writes Key: FLINK-24436 URL: https://issues.apache.org/jira/browse/FLINK-24436 Project: Flink Issue Type: Bug Reporter: Yuan Mei -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: The Apache Flink should pay more attention to ensuring API compatibility.
Hi all, Nice thread and great discussion! Ecosystem is one of the most important things to the Flink community, we should pay more attention to API compatibility. Marking all connector APIs @Public is a good idea, not only mark the Table/SQL connector APIs public, but also the new Source/Sink APIs as public. Besides, we should also add a check item to the Verify Release documentation[1] to verify the release is backward-compatible for connectors. From my point of view, such backward incompatibility should cancel the vote. Regarding the SourceReaderContext#metricGroup compatibility problem in 1.14.0, I would suggest starting a new discussion thread to see whether we have any idea to fix it. We should fix it ASAP! Otherwise iceberg/hudi/cdc communities will get frustrated again when upgrading to 1.14. Maybe we still have time to release a minor version, because there is no connector upgraded to 1.14.0 yet. What do you think? @Leonard Xu @Arvid Heise @Piotr Nowojski Best, Jark [1]: https://cwiki.apache.org/confluence/display/FLINK/Verifying+a+Flink+Release On Wed, 29 Sept 2021 at 09:46, OpenInx wrote: > > Apart from this being `@PublicEvolving` > > From my perspective, annotating the 'DynamicTableSink' to be a > 'PublicEvolving' class is not reasonable, because that means devs could > just change the basic API which all downstream connectors are depending on > easily when iterating flink from 1.12 to 1.13 (according to the wiki [1]). > This implies all downstream maintainers must take on this maintenance > burden, and it also makes our flink ecosystem very fragile. Changing the > 'DynamicTableSink' between two major versions sounds reasonable to me, but > unreasonable for uncompatibility changes between two minor versions. I > think we may need to check those API which are annotated 'PublicEnvoling' > while should be 'Public' because of the dependency from all connectors. > We should ensure the stability of those APIs that are necessary to > implement the connector, and at the same time implement the updated v2 > version of the API. After all v2 APIs are considered stable, we will mark > them as stable. Instead of releasing a version of the API, some of the APIs > necessary to implement the connector are marked as stable and some are > marked as unstable, which is very unfriendly to downstream. Because > downstream essentially every upgrade requires refactoring of the code. > > > We are trying to provide forward compatibility: applications using > `@Public` APIs compiled against Flink 1.12.x, should work fine in Flink > 1.13.x > > Thanks for clarifying this. Sounds reasonable to me, then we apache > iceberg could just use flink 1.12.x to build the flink+iceberg connector > and should make all the tests work fine for both flink 1.12 & flink 1.13. > For the `ResolvedCatalogTable` changes, I don't think it guarantees the > forward compatibility as you said, because the flink-iceberg-runtime.jar > compiled by flink 1.12 can still not works fine in flink 1.13 because it > will need the flink1.12's `CatalogTable` to be cast to a flink1.13 's > `ResolvedCatalogTable`. In general, I agree that forward compatibility is > a more clear compatibility guarantee. > > [1]. > https://cwiki.apache.org/confluence/display/FLINK/Stability+Annotations > > > On Tue, Sep 28, 2021 at 10:33 PM David Morávek wrote: > > > > > > > I think we have a compile time checks for breaking changes in `@Public` > > > marked classes/interfaces using japicmp [1]. > > > > > > Nice, thanks for pointing that out, I'll take a closer look at it ;) > > > > Best, > > D. > > > > On Tue, Sep 28, 2021 at 4:14 PM Piotr Nowojski > > wrote: > > > > > > - We don't have any safeguards for stable API breaks. Big +1 for > Ingo's > > > effort with architectural tests [3]. > > > > > > I think we have a compile time checks for breaking changes in `@Public` > > > marked classes/interfaces using japicmp [1]. > > > > > > Piotrek > > > > > > [1] https://github.com/apache/flink/blob/master/pom.xml#L2014:L2084 > > > > > > wt., 28 wrz 2021 o 15:59 David Morávek napisał(a): > > > > > > > This is a super interesting topic and there is already a great > > > discussion. > > > > Here are few thoughts: > > > > > > > > - There is a delicate balance between fast delivery of the new > features > > > and > > > > API stability. Even though we should be careful with breaking > evolving > > > > interfaces, it shouldn't stop us from making fast progress / iterate > on > > > > features. > > > > - There are two camps of users. One camp prefers more frequent > > releases / > > > > new features (early adopters) and second that prefer less frequent > > stable > > > > releases. There was already a great discussion about this at Flink > 1.14 > > > > thread [1]. > > > > - We're already trying to be explicit about which API's may break via > > > > annotations and the feature radar [2]. Stability annotations are a > well > > > > known concept used by many projects. I thi
[jira] [Created] (FLINK-24437) Remove unhandled exception handler from CuratorFramework before closing it
Till Rohrmann created FLINK-24437:
-
Summary: Remove unhandled exception handler from CuratorFramework
before closing it
Key: FLINK-24437
URL: https://issues.apache.org/jira/browse/FLINK-24437
Project: Flink
Issue Type: Improvement
Components: Runtime / Coordination
Affects Versions: 1.14.0, 1.15.0
Reporter: Till Rohrmann
Fix For: 1.15.0, 1.14.1
With FLINK-24021 we add an unhandled exception handler to the started
{{CuratorFramework}}. In order to avoid that shutting down the
{{CuratorFramework}} causes triggering of this handler, we should unregister it
before closing the {{CuratorFramework}} instance.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
Re: The Apache Flink should pay more attention to ensuring API compatibility.
Hi, > [...] but also the new Source/Sink APIs as public I'm not really involved in the new Source/Sink APIs and will happily listen to the developers working with them here, but since they are new, we should also be careful not to mark them as stable too quickly. We've only begun updating the existing connectors to these interfaces at the moment. Making more progress here and keeping new APIs as Evolving for a couple of minor releases is probably still a good idea. Maybe we should even have actual rules on when APIs can/should be promoted? More actively checking backwards-compatibility during a release sounds like a great idea regardless, of course. Ingo On Fri, Oct 1, 2021 at 9:19 AM Jark Wu wrote: > Hi all, > > Nice thread and great discussion! Ecosystem is one of the most important > things > to the Flink community, we should pay more attention to API compatibility. > > Marking all connector APIs @Public is a good idea, not only mark the > Table/SQL > connector APIs public, but also the new Source/Sink APIs as public. > Besides, we should also add a check item to the Verify Release > documentation[1] > to verify the release is backward-compatible for connectors. From my point > of view, > such backward incompatibility should cancel the vote. > > Regarding the SourceReaderContext#metricGroup compatibility problem in > 1.14.0, I would > suggest starting a new discussion thread to see whether we have any idea to > fix it. We should > fix it ASAP! Otherwise iceberg/hudi/cdc communities will get frustrated > again when upgrading > to 1.14. Maybe we still have time to release a minor version, because > there is no > connector upgraded to 1.14.0 yet. What do you think? @Leonard Xu > @Arvid Heise @Piotr Nowojski > > > Best, > Jark > > [1]: > https://cwiki.apache.org/confluence/display/FLINK/Verifying+a+Flink+Release > > On Wed, 29 Sept 2021 at 09:46, OpenInx wrote: > > > > Apart from this being `@PublicEvolving` > > > > From my perspective, annotating the 'DynamicTableSink' to be a > > 'PublicEvolving' class is not reasonable, because that means devs could > > just change the basic API which all downstream connectors are depending > on > > easily when iterating flink from 1.12 to 1.13 (according to the wiki > [1]). > > This implies all downstream maintainers must take on this maintenance > > burden, and it also makes our flink ecosystem very fragile. Changing > the > > 'DynamicTableSink' between two major versions sounds reasonable to me, > but > > unreasonable for uncompatibility changes between two minor versions. I > > think we may need to check those API which are annotated 'PublicEnvoling' > > while should be 'Public' because of the dependency from all connectors. > > We should ensure the stability of those APIs that are necessary to > > implement the connector, and at the same time implement the updated v2 > > version of the API. After all v2 APIs are considered stable, we will mark > > them as stable. Instead of releasing a version of the API, some of the > APIs > > necessary to implement the connector are marked as stable and some are > > marked as unstable, which is very unfriendly to downstream. Because > > downstream essentially every upgrade requires refactoring of the code. > > > > > We are trying to provide forward compatibility: applications using > > `@Public` APIs compiled against Flink 1.12.x, should work fine in Flink > > 1.13.x > > > > Thanks for clarifying this. Sounds reasonable to me, then we apache > > iceberg could just use flink 1.12.x to build the flink+iceberg connector > > and should make all the tests work fine for both flink 1.12 & flink 1.13. > > For the `ResolvedCatalogTable` changes, I don't think it guarantees the > > forward compatibility as you said, because the flink-iceberg-runtime.jar > > compiled by flink 1.12 can still not works fine in flink 1.13 because it > > will need the flink1.12's `CatalogTable` to be cast to a flink1.13 's > > `ResolvedCatalogTable`. In general, I agree that forward compatibility > is > > a more clear compatibility guarantee. > > > > [1]. > > https://cwiki.apache.org/confluence/display/FLINK/Stability+Annotations > > > > > > On Tue, Sep 28, 2021 at 10:33 PM David Morávek wrote: > > > > > > > > > > I think we have a compile time checks for breaking changes in > `@Public` > > > > marked classes/interfaces using japicmp [1]. > > > > > > > > > Nice, thanks for pointing that out, I'll take a closer look at it ;) > > > > > > Best, > > > D. > > > > > > On Tue, Sep 28, 2021 at 4:14 PM Piotr Nowojski > > > wrote: > > > > > > > > - We don't have any safeguards for stable API breaks. Big +1 for > > Ingo's > > > > effort with architectural tests [3]. > > > > > > > > I think we have a compile time checks for breaking changes in > `@Public` > > > > marked classes/interfaces using japicmp [1]. > > > > > > > > Piotrek > > > > > > > > [1] https://github.com/apache/flink/blob/master/pom.xml#L2014:L2084 > > > > > > > > wt., 28 wrz 2021
Re: [Flink blogs]
Hi all, Thanks a lot for your feedback guys ! Special thanks to Fabian, Till and Arvid (in a private discussion) ! The consensus seems to go toward the blog post on migrating a batch pipeline from DataSet API to DataStream API. For the record it is linked to a work I did lately (unfortunately not public, let's see if I can make it public in the future) of testing TPCDS performance framework on Flink. I know there is an impl already in the repo using flink-SQL but I wanted to implement it lower level using DataSet API and later DataStream API. It uses parquet (so old format). The query I implemented is TPCDS Query 3. That is for the use case of this future blob post. Indeed, as Fabian and Till said, it can easily become a serie. Second blog to receive lower consensus: manual join with KeyedCoProcessFunction in DataStream (thanks Till !). I will off course add a pointer to the new target for users :Table/SQL API as reminded by Fabian. Another blog post could be related to performances: during this bench, I have observed the cost of SQL translation compared to lower level, the improvement of perfs in DataStream or the improvement of perfs entailed by Blink planner. That also could be a good blog post. Also I tend not to compare perfs with other apache big data projects such as Spark because they all have their strengths, their tricky parameters and in the end we often end up comparing not 100% comparable things. Regarding the other topics, as I wrote, I was doubting that they could have interest mainly because of deprecation of formats, steering users toward Table/SQL API or because of too low level topics. Thanks for confirming my doubts ! Best Etienne On 30/09/2021 15:51, Till Rohrmann wrote: Hi Etienne, Great to see that you want to write about one of the topics you have worked on! Spreading the word about changes/improvements/new features is always super important. As a general recommendation I think you should write about the topic you are most excited about. This usually results in an interesting blog post for others. If you have multiple favourites, then I would think about what topic could be most interesting for users. In my experience, blog posts that deal with a user problem (e.g. how to solve xyz) get more attention than technical blog posts. Having said this, I think the following topics would be good candidates: - migration of pipelines from DataSet API to DataStream API As Fabian said, this could easily become a series of blog posts. Maybe this could also become part of the documentation. - doing a manual join in DataStream API in batch mode with /KeyedCoProcessFunction I could see that this is a nice blog post about a concrete recipe on how to solve a certain set of problems with the DataStream API. Fabian is right that in the future we will try to steer people towards the Table API but maybe the join condition cannot be easily expressed with SQL so that people would naturally switch to the DataStream API for it. - back pressure in checkpointing Improving the understanding of Flink operations is always a good and worthwhile idea imo. Cheers, Till On Thu, Sep 30, 2021 at 10:19 AM Fabian Paul wrote: Hi Etienne, Thanks for reaching out I think your list already looks very appealing. * - metrics (https://github.com/apache/flink/pull/14510): it was dealing with delimiters. I think it is a bit low level for a blog post ? * I am also unsure whether this a good fit to present. I can only imagine showing what kind of use-case it supports. * - migration of pipelines from DataSet API to DataStream API: it is already discussed in the flink website * This is definitely something I’d like to see in my opinion it can also become a series because the topic has a lot of aspects. If you want to write a post about it it would be great to show the migration of a more complex pipeline (i.e. old formats, incompatible types ….). Many users will eventually face this so it has a big impact. FYI probably only Flink 1.13 is the latest version with full DataSet support. * - accumulators (https://github.com/apache/flink/pull/14558): it was about an asynchronous get, once again a bit too low level for a blog post ? * To me accumulator are a kind of internal concept but maybe you can provide the use-case which drove this change? Probably explaining the semantics of them is already complicated. * - FileInputFormat mainly parquet improvements and fixes (https://github.com/apache/flink/pull/15725, https://github.com/apache/flink/pull/15172, https://github.com/apache/flink/pull/15156): interesting but as this API is being decommissioned, it might not be a good subject ? * You have already summarized it: it is being deprecated and a much more interesting topic is the migration from DataSet to the DataStream API in case these old formats are used. * - doing a manual join in DataStream API in batch mode with /KeyedCoProcessFunction///
[jira] [Created] (FLINK-24438) Port Kinesis Source to new Source API (FLIP-27)
Martijn Visser created FLINK-24438: -- Summary: Port Kinesis Source to new Source API (FLIP-27) Key: FLINK-24438 URL: https://issues.apache.org/jira/browse/FLINK-24438 Project: Flink Issue Type: Improvement Components: Connectors / Kinesis Reporter: Martijn Visser We want to port the Kinesis Source to the new Source API (FLIP-27) and deprecate the old FlinkKinesisConsumer -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: The Apache Flink should pay more attention to ensuring API compatibility.
The issue is that if we do not mark them as Public, we will always have incompatibilities. The change of SourceReaderContext#metricGroup is perfectly fine according to the annotation. The implications that we see here just mean that the interfaces have been expected to be Public. And now the question is what do we expect? Should we expect connector devs to release different connector binaries for different Flink minors? Then PublicEvolving is fine. If we expect that the same connector can work across multiple Flink versions, we need to go into Public. It doesn't make sense to keep them PublicEvolving on the annotation but implicitly assume them to be Public. @Jark Wu I don't see a way to revert the change of SourceReaderContext#metricGroup. For now, connector devs that expose metrics need to release 2 versions. If we change it back, then a specific connector would work for 1.14.1 and 1.13.X but not for 1.14.0 and this would be even more confusing. On Fri, Oct 1, 2021 at 10:49 AM Ingo Bürk wrote: > Hi, > > > [...] but also the new Source/Sink APIs as public > > I'm not really involved in the new Source/Sink APIs and will happily > listen to the developers working with them here, but since they are new, we > should also be careful not to mark them as stable too quickly. We've only > begun updating the existing connectors to these interfaces at the moment. > Making more progress here and keeping new APIs as Evolving for a couple of > minor releases is probably still a good idea. Maybe we should even have > actual rules on when APIs can/should be promoted? > > More actively checking backwards-compatibility during a release sounds > like a great idea regardless, of course. > > > Ingo > > On Fri, Oct 1, 2021 at 9:19 AM Jark Wu wrote: > >> Hi all, >> >> Nice thread and great discussion! Ecosystem is one of the most important >> things >> to the Flink community, we should pay more attention to API compatibility. >> >> Marking all connector APIs @Public is a good idea, not only mark the >> Table/SQL >> connector APIs public, but also the new Source/Sink APIs as public. >> Besides, we should also add a check item to the Verify Release >> documentation[1] >> to verify the release is backward-compatible for connectors. From my point >> of view, >> such backward incompatibility should cancel the vote. >> >> Regarding the SourceReaderContext#metricGroup compatibility problem in >> 1.14.0, I would >> suggest starting a new discussion thread to see whether we have any idea >> to >> fix it. We should >> fix it ASAP! Otherwise iceberg/hudi/cdc communities will get frustrated >> again when upgrading >> to 1.14. Maybe we still have time to release a minor version, because >> there is no >> connector upgraded to 1.14.0 yet. What do you think? @Leonard Xu >> @Arvid Heise @Piotr Nowojski >> >> >> Best, >> Jark >> >> [1]: >> >> https://cwiki.apache.org/confluence/display/FLINK/Verifying+a+Flink+Release >> >> On Wed, 29 Sept 2021 at 09:46, OpenInx wrote: >> >> > > Apart from this being `@PublicEvolving` >> > >> > From my perspective, annotating the 'DynamicTableSink' to be a >> > 'PublicEvolving' class is not reasonable, because that means devs could >> > just change the basic API which all downstream connectors are depending >> on >> > easily when iterating flink from 1.12 to 1.13 (according to the wiki >> [1]). >> > This implies all downstream maintainers must take on this maintenance >> > burden, and it also makes our flink ecosystem very fragile. Changing >> the >> > 'DynamicTableSink' between two major versions sounds reasonable to me, >> but >> > unreasonable for uncompatibility changes between two minor versions. I >> > think we may need to check those API which are annotated >> 'PublicEnvoling' >> > while should be 'Public' because of the dependency from all connectors. >> > We should ensure the stability of those APIs that are necessary to >> > implement the connector, and at the same time implement the updated v2 >> > version of the API. After all v2 APIs are considered stable, we will >> mark >> > them as stable. Instead of releasing a version of the API, some of the >> APIs >> > necessary to implement the connector are marked as stable and some are >> > marked as unstable, which is very unfriendly to downstream. Because >> > downstream essentially every upgrade requires refactoring of the code. >> > >> > > We are trying to provide forward compatibility: applications using >> > `@Public` APIs compiled against Flink 1.12.x, should work fine in Flink >> > 1.13.x >> > >> > Thanks for clarifying this. Sounds reasonable to me, then we apache >> > iceberg could just use flink 1.12.x to build the flink+iceberg connector >> > and should make all the tests work fine for both flink 1.12 & flink >> 1.13. >> > For the `ResolvedCatalogTable` changes, I don't think it guarantees the >> > forward compatibility as you said, because the flink-iceberg-runtime.jar >> > compiled by flink 1.12 can still not works fine
Re: The Apache Flink should pay more attention to ensuring API compatibility.
Hi Arvid, > Should we expect connector devs to release different connector binaries for different Flink minors? >From the discussion of this thread, I think the answer is obviously "not", otherwise OpenInx won't start this discussion. As a maintainer of flink-cdc-connector, I have to say that it's very hard to release connectors for different flink versions. Usually, the connector community doesn't have so much time to maintain different branches/modules/code for different flink versions. > If we change it back, then a specific connector would work for 1.14.1 and 1.13.X but not for 1.14.0 and this would be even more confusing. I think this is fine. IMO, this is a blocker issue of 1.14.0 which breaks Source connectors. We should suggest users to use 1.14.1 if they use Source connectors. Best, Jark On Fri, 1 Oct 2021 at 19:05, Arvid Heise wrote: > The issue is that if we do not mark them as Public, we will always have > incompatibilities. The change of SourceReaderContext#metricGroup is > perfectly fine according to the annotation. The implications that we see > here just mean that the interfaces have been expected to be Public. > > And now the question is what do we expect? > Should we expect connector devs to release different connector binaries > for different Flink minors? Then PublicEvolving is fine. > If we expect that the same connector can work across multiple Flink > versions, we need to go into Public. > > It doesn't make sense to keep them PublicEvolving on the annotation but > implicitly assume them to be Public. > > @Jark Wu I don't see a way to revert the change of > SourceReaderContext#metricGroup. For now, connector devs that expose > metrics need to release 2 versions. If we change it back, then a specific > connector would work for 1.14.1 and 1.13.X but not for 1.14.0 and this > would be even more confusing. > > On Fri, Oct 1, 2021 at 10:49 AM Ingo Bürk wrote: > >> Hi, >> >> > [...] but also the new Source/Sink APIs as public >> >> I'm not really involved in the new Source/Sink APIs and will happily >> listen to the developers working with them here, but since they are new, we >> should also be careful not to mark them as stable too quickly. We've only >> begun updating the existing connectors to these interfaces at the moment. >> Making more progress here and keeping new APIs as Evolving for a couple of >> minor releases is probably still a good idea. Maybe we should even have >> actual rules on when APIs can/should be promoted? >> >> More actively checking backwards-compatibility during a release sounds >> like a great idea regardless, of course. >> >> >> Ingo >> >> On Fri, Oct 1, 2021 at 9:19 AM Jark Wu wrote: >> >>> Hi all, >>> >>> Nice thread and great discussion! Ecosystem is one of the most important >>> things >>> to the Flink community, we should pay more attention to API >>> compatibility. >>> >>> Marking all connector APIs @Public is a good idea, not only mark the >>> Table/SQL >>> connector APIs public, but also the new Source/Sink APIs as public. >>> Besides, we should also add a check item to the Verify Release >>> documentation[1] >>> to verify the release is backward-compatible for connectors. From my >>> point >>> of view, >>> such backward incompatibility should cancel the vote. >>> >>> Regarding the SourceReaderContext#metricGroup compatibility problem in >>> 1.14.0, I would >>> suggest starting a new discussion thread to see whether we have any idea >>> to >>> fix it. We should >>> fix it ASAP! Otherwise iceberg/hudi/cdc communities will get frustrated >>> again when upgrading >>> to 1.14. Maybe we still have time to release a minor version, because >>> there is no >>> connector upgraded to 1.14.0 yet. What do you think? @Leonard Xu >>> @Arvid Heise @Piotr Nowojski >>> >>> >>> Best, >>> Jark >>> >>> [1]: >>> >>> https://cwiki.apache.org/confluence/display/FLINK/Verifying+a+Flink+Release >>> >>> On Wed, 29 Sept 2021 at 09:46, OpenInx wrote: >>> >>> > > Apart from this being `@PublicEvolving` >>> > >>> > From my perspective, annotating the 'DynamicTableSink' to be a >>> > 'PublicEvolving' class is not reasonable, because that means devs could >>> > just change the basic API which all downstream connectors are >>> depending on >>> > easily when iterating flink from 1.12 to 1.13 (according to the wiki >>> [1]). >>> > This implies all downstream maintainers must take on this maintenance >>> > burden, and it also makes our flink ecosystem very fragile. Changing >>> the >>> > 'DynamicTableSink' between two major versions sounds reasonable to me, >>> but >>> > unreasonable for uncompatibility changes between two minor versions. >>> I >>> > think we may need to check those API which are annotated >>> 'PublicEnvoling' >>> > while should be 'Public' because of the dependency from all >>> connectors. >>> > We should ensure the stability of those APIs that are necessary to >>> > implement the connector, and at the same time implement the updated v2 >>> > vers
[jira] [Created] (FLINK-24440) Announce and combine latest watermarks across SourceReaders
Piotr Nowojski created FLINK-24440: -- Summary: Announce and combine latest watermarks across SourceReaders Key: FLINK-24440 URL: https://issues.apache.org/jira/browse/FLINK-24440 Project: Flink Issue Type: Sub-task Components: Connectors / Common Reporter: Piotr Nowojski Assignee: Piotr Nowojski Fix For: 1.15.0 # Each SourceReader should inform it's SourceCoordinator about the latest watermark that it has emitted so far # SourceCoordinators should combine those watermarks and broadcast the aggregated result back to all SourceReaders -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-24441) Block SourceReader when watermarks are out of alignment
Piotr Nowojski created FLINK-24441: -- Summary: Block SourceReader when watermarks are out of alignment Key: FLINK-24441 URL: https://issues.apache.org/jira/browse/FLINK-24441 Project: Flink Issue Type: Sub-task Components: Connectors / Common Reporter: Piotr Nowojski Assignee: Piotr Nowojski Fix For: 1.15.0 SourceReader should become unavailable once it's latest watermark is too far into the future -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-24439) Introduce a CoordinatorStore
Piotr Nowojski created FLINK-24439:
--
Summary: Introduce a CoordinatorStore
Key: FLINK-24439
URL: https://issues.apache.org/jira/browse/FLINK-24439
Project: Flink
Issue Type: Sub-task
Components: Connectors / Common
Reporter: Piotr Nowojski
Assignee: Piotr Nowojski
Fix For: 1.15.0
In order to allow {{SourceCoordinator}}s from different {{Source}}s (for
example two different Kafka sources, or Kafka and Kinesis) to align watermarks,
they have to be able to exchange information/aggregate watermarks from those
different Sources. To enable this, we need to provide some {{CoordinatorStore}}
concept, that would be a thread safe singleton.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
Re: The Apache Flink should pay more attention to ensuring API compatibility.
Hi, I don't understand why we are talking about this being a blocker issue? New sources were not marked as @Public for a good reason until 1.14. I agree, we should try better at making APIs @Public sooner. I was even proposing to create strict timeout rules (enforced via some automated checks) like (unless for a very very good reason) everything marked @PublicEvolving or @Experimental should be upgraded to @Public after for example 2 years [1]. But for example the new Sink API IMO is too fresh to make it `@Public`. It doesn't change the fact that if we could provide a compatibility layer between 1.13.x and 1.14.x for this SourceReaderContext issue, it would be a nice thing to do. I would be -1 for keeping it forever, but trying to support forward compatibility of `@PublicEvolving` APIs for one or two releases into the future might be a good rule of thumb. Best, Piotrek [1] "[DISCUSS] Dealing with deprecated and legacy code in Flink" on the dev mailing list pt., 1 paź 2021 o 16:56 Jark Wu napisał(a): > Hi Arvid, > > > Should we expect connector devs to release different connector binaries > for different Flink minors? > From the discussion of this thread, I think the answer is obviously "not", > otherwise OpenInx won't start > this discussion. As a maintainer of flink-cdc-connector, I have to say > that it's very hard to release > connectors for different flink versions. Usually, the connector community > doesn't have so much time to > maintain different branches/modules/code for different flink versions. > > > If we change it back, then a specific connector would work for 1.14.1 and > 1.13.X but not for 1.14.0 and this would be even more confusing. > I think this is fine. IMO, this is a blocker issue of 1.14.0 which breaks > Source connectors. > We should suggest users to use 1.14.1 if they use Source connectors. > > Best, > Jark > > > On Fri, 1 Oct 2021 at 19:05, Arvid Heise wrote: > > > The issue is that if we do not mark them as Public, we will always have > > incompatibilities. The change of SourceReaderContext#metricGroup is > > perfectly fine according to the annotation. The implications that we see > > here just mean that the interfaces have been expected to be Public. > > > > And now the question is what do we expect? > > Should we expect connector devs to release different connector binaries > > for different Flink minors? Then PublicEvolving is fine. > > If we expect that the same connector can work across multiple Flink > > versions, we need to go into Public. > > > > It doesn't make sense to keep them PublicEvolving on the annotation but > > implicitly assume them to be Public. > > > > @Jark Wu I don't see a way to revert the change of > > SourceReaderContext#metricGroup. For now, connector devs that expose > > metrics need to release 2 versions. If we change it back, then a specific > > connector would work for 1.14.1 and 1.13.X but not for 1.14.0 and this > > would be even more confusing. > > > > On Fri, Oct 1, 2021 at 10:49 AM Ingo Bürk wrote: > > > >> Hi, > >> > >> > [...] but also the new Source/Sink APIs as public > >> > >> I'm not really involved in the new Source/Sink APIs and will happily > >> listen to the developers working with them here, but since they are > new, we > >> should also be careful not to mark them as stable too quickly. We've > only > >> begun updating the existing connectors to these interfaces at the > moment. > >> Making more progress here and keeping new APIs as Evolving for a couple > of > >> minor releases is probably still a good idea. Maybe we should even have > >> actual rules on when APIs can/should be promoted? > >> > >> More actively checking backwards-compatibility during a release sounds > >> like a great idea regardless, of course. > >> > >> > >> Ingo > >> > >> On Fri, Oct 1, 2021 at 9:19 AM Jark Wu wrote: > >> > >>> Hi all, > >>> > >>> Nice thread and great discussion! Ecosystem is one of the most > important > >>> things > >>> to the Flink community, we should pay more attention to API > >>> compatibility. > >>> > >>> Marking all connector APIs @Public is a good idea, not only mark the > >>> Table/SQL > >>> connector APIs public, but also the new Source/Sink APIs as public. > >>> Besides, we should also add a check item to the Verify Release > >>> documentation[1] > >>> to verify the release is backward-compatible for connectors. From my > >>> point > >>> of view, > >>> such backward incompatibility should cancel the vote. > >>> > >>> Regarding the SourceReaderContext#metricGroup compatibility problem in > >>> 1.14.0, I would > >>> suggest starting a new discussion thread to see whether we have any > idea > >>> to > >>> fix it. We should > >>> fix it ASAP! Otherwise iceberg/hudi/cdc communities will get frustrated > >>> again when upgrading > >>> to 1.14. Maybe we still have time to release a minor version, because > >>> there is no > >>> connector upgraded to 1.14.0 yet. What do you think? @Leonard Xu > >>> @Arvid Heise @Piotr Nowoj
