Stephan, thanks a lot for explaining the file connector. that makes sense. I was asking because we were trying to reuse some of the implementations in the file source for Iceberg source. Flink Iceberg source lives in the Iceberg repo, which is not possible to code against the master branch of the Flink code.
On Mon, Nov 2, 2020 at 3:31 AM Stephan Ewen <se...@apache.org> wrote: > Hi Steven! > > So far there are no plans to pick back the file system connector code. This > is still evolving and not finalized for 1.12, so I don't feel it is a good > candidate to be backported. > However, with the base connector changes backported, you should be able to > run the file connector code from master against 1.11.3. > > The collect() utils can be picked back, I see no issue with that (it is > isolated utilities). > > Best, > Stephan > > > On Mon, Nov 2, 2020 at 3:02 AM Steven Wu <stevenz...@gmail.com> wrote: > > > Basically, it would be great to get the latest code in the > > flink-connector-files (FLIP-27). > > > > On Sat, Oct 31, 2020 at 9:57 AM Steven Wu <stevenz...@gmail.com> wrote: > > > > > Stephan, it will be great if we can also backport the DataStreamUtils > > > related commits that help with collecting output from unbounded > streams. > > > e.g. > > > > > > > > > > > > https://github.com/apache/flink/commit/09a7a66b7313fea64817fe960a8da1265b428efc > > > > > > I tried to copy and paste the code to unblock myself. but it quickly > got > > > into the rabbit hole of more and more code. > > > > > > On Fri, Oct 30, 2020 at 11:02 AM Stephan Ewen <se...@apache.org> > wrote: > > > > > >> I have started with backporting the source API changes. Some minor > > >> conflicts to solve, will need a bit more to finish this. > > >> > > >> On Fri, Oct 30, 2020 at 7:25 AM Tzu-Li (Gordon) Tai < > > tzuli...@apache.org> > > >> wrote: > > >> > > >> > @Stephan Ewen <se...@apache.org> > > >> > Are there already plans or ongoing efforts for backporting the list > of > > >> > FLIP-27 changes that you posted? > > >> > > > >> > On Thu, Oct 29, 2020 at 7:08 PM Xintong Song <tonysong...@gmail.com > > > > >> > wrote: > > >> > > > >> >> Hi folks, > > >> >> > > >> >> Just to provide some updates concerning the status on the > > >> >> test instabilities. > > >> >> > > >> >> Currently, we have 30 unresolved tickets labeled with `Affects > > Version` > > >> >> 1.11.x. > > >> >> > > >> >> > > >> > > > https://issues.apache.org/jira/browse/FLINK-19775?filter=12348580&jql=project%20%3D%20FLINK%20AND%20resolution%20%3D%20Unresolved%20AND%20affectedVersion%20in%20(1.11.0%2C%201.11.1%2C%201.11.2%2C%201.11.3)%20AND%20labels%20%3D%20test-stability%20ORDER%20BY%20created%20DESC > > >> >> > > >> >> Among the 30 tickets, 11 of them are: > > >> >> - Have occured in the recent 3 months > > >> >> - Not confirmed to be pure testability issues > > >> >> - Not confirmed to be rare condition cases > > >> >> > > >> >> It would be nice if someone familiar with these components can > take a > > >> look > > >> >> into these issues. > > >> >> > > >> >> - https://issues.apache.org/jira/browse/FLINK-17159 (ES6) > > >> >> - https://issues.apache.org/jira/browse/FLINK-17912 (Kafka) > > >> >> - https://issues.apache.org/jira/browse/FLINK-17949 (Kafka) > > >> >> ⁃ https://issues.apache.org/jira/browse/FLINK-18444 (Kafka) > > >> >> - https://issues.apache.org/jira/browse/FLINK-18634 (Kafka) > > >> >> - https://issues.apache.org/jira/browse/FLINK-18648 (Kafka) > > >> >> - https://issues.apache.org/jira/browse/FLINK-18807 (Kafka) > > >> >> - https://issues.apache.org/jira/browse/FLINK-19369 > (BlobClientTest) > > >> >> - https://issues.apache.org/jira/browse/FLINK-19436 (TPCDS) > > >> >> - https://issues.apache.org/jira/browse/FLINK-19690 > (Format/Parquet) > > >> >> - https://issues.apache.org/jira/browse/FLINK-19775 > > >> >> (SystemProcessingTimeServiceTest) > > >> >> > > >> >> Thank you~ > > >> >> > > >> >> Xintong Song > > >> >> > > >> >> > > >> >> > > >> >> On Thu, Oct 29, 2020 at 10:21 AM Jingsong Li < > jingsongl...@gmail.com > > > > > >> >> wrote: > > >> >> > > >> >> > +1 to backport the FLIP-27 adjustments to 1.11.x. > > >> >> > > > >> >> > If possible, that would be great. Many people are looking forward > > to > > >> the > > >> >> > FLIP-27 interface, but they don't want to take the risk to > upgrade > > to > > >> >> 1.12 > > >> >> > (And wait 1.12). After all, 1.11 is a relatively stable version. > > >> >> > > > >> >> > Best, > > >> >> > Jingsong > > >> >> > > > >> >> > On Thu, Oct 29, 2020 at 1:24 AM Stephan Ewen <se...@apache.org> > > >> wrote: > > >> >> > > > >> >> > > Thanks for starting this. > > >> >> > > > > >> >> > > +1 form my side to backport the FLIP-27 adjustments to 1.11.x. > > >> >> > > > > >> >> > > There were quite a few changes, and I think we need to > > cherry-pick > > >> >> them > > >> >> > all > > >> >> > > to not get some inconsistent mix of changes and many merge > > >> conflicts. > > >> >> > > I made a list below of what we need to add to "release-1.11". > > >> >> > > > > >> >> > > *#### Core Source API Changes to backport (in REVERSE order)* > > >> >> > > > > >> >> > > (Use: "git log > > >> >> > > > flink-core/src/main/java/org/apache/flink/api/connector/source") > > >> >> > > > > >> >> > > commit 162c072e9265a7b6dd9d6f5459eb7974091c4c4e > > >> >> > > [FLINK-19492][core] Consolidate Source Events between Source > API > > >> and > > >> >> > Split > > >> >> > > Reader API > > >> >> > > > > >> >> > > commit 428fbbad85379f71a9fbf1a1ab3e3dcfa9006436 > > >> >> > > [FLINK-19205][core] Add access to configuration and hostname in > > the > > >> >> > > SourceReaderContext > > >> >> > > > > >> >> > > commit 723e1790744ddc78d19c7c978442af1383f38d33 > > >> >> > > [hotfix][core] Add to Source Enumerator convenience methods to > > >> assign > > >> >> > > single split > > >> >> > > > > >> >> > > *#### Connector Base Changes to Backport (in REVERSE order)* > > >> >> > > > > >> >> > > (Use: "git log flink-connectors/flink-connector-base") > > >> >> > > > > >> >> > > commit 401f56fe9d6b0271260edf9787cdcbfe4d03874d > > >> >> > > [FLINK-19427][FLINK-19489][tests] Fix test conditions for > > >> >> > > 'SplitFetcherTest.testNotifiesWhenGoingIdleConcurrent()' > > >> >> > > > > >> >> > > commit 68c5c2ff779d82a1ff81ffaf60d8a1b283797db1 > > >> >> > > [FLINK-19448][connector base] Explicitly check for un-expected > > >> >> condition > > >> >> > > that would leave an inconsistent state > > >> >> > > > > >> >> > > commit 162c072e9265a7b6dd9d6f5459eb7974091c4c4e > > >> >> > > [FLINK-19492][core] Consolidate Source Events between Source > API > > >> and > > >> >> > Split > > >> >> > > Reader API > > >> >> > > > > >> >> > > commit c1ca7a4c7c21ec8868c14cdddd43c559625b794c > > >> >> > > [refactor][tests] Move some source test utils from > > >> >> flink-connector-base > > >> >> > to > > >> >> > > flink-core > > >> >> > > > > >> >> > > commit ee5c4c211c35c70d28252363bbc8400453609977 > > >> >> > > [FLINK-19251][connectors] Avoid confusing queue handling in > > >> >> > > "SplitReader.handleSplitsChanges()" > > >> >> > > > > >> >> > > commit 5abef56b2bf85bcac786f6b16b6899b6cced7176 > > >> >> > > [FLINK-19250][connectors] Fix error propagation in connector > base > > >> >> > > (SplitFetcherManager). > > >> >> > > > > >> >> > > commit 8fcca837c55a9216595ee4c03038b52747098dbb > > >> >> > > [hotfix][connectors] Improve JavaDocs for > > >> SingleThreadFetcherManager > > >> >> > > > > >> >> > > commit 4700bb5dde3303cbe98882f6beb7379425717b01 > > >> >> > > [FLINK-19225][connectors] Various small improvements to > > >> >> SourceReaderBase > > >> >> > > (part 2) > > >> >> > > > > >> >> > > commit 12261c6b7ed6478a9b9f6a69cb58246b83cab9b7 > > >> >> > > [FLINK-17393][connectors] (follow-up) Wakeup the SplitFetchers > > more > > >> >> > > elegantly. > > >> >> > > > > >> >> > > commit c60aaff0249bfd6b5871b7f82e03efc487a54d6b > > >> >> > > [hotfix][tests] Extend test coverage for > > >> FutureCompletingBlockingQueue > > >> >> > > > > >> >> > > commit cef8a587d7fd2fe64cc644da5ed095d82e46f631 > > >> >> > > [FLINK-19245][connectors] Set default capacity for > > >> >> > > FutureCompletingBlockingQueue. > > >> >> > > > > >> >> > > commit 4ea95782b4c6a2538153d4d16ad3f4839c7de0fb > > >> >> > > [FLINK-19223][connectors] Simplify Availability Future Model in > > >> Base > > >> >> > > Connector > > >> >> > > > > >> >> > > commit 511857049ba30c8ff0ee56da551fa4a479dc583e > > >> >> > > [FLINK-18128][connectors] Ensure idle split fetchers lead to > > >> >> availability > > >> >> > > notifications. > > >> >> > > > > >> >> > > commit a8206467af0830dcb89623ea068b5ca3b3450c92 > > >> >> > > [refactor][core] Eagerly initialize the FetchTask to support > > proper > > >> >> unit > > >> >> > > testing > > >> >> > > > > >> >> > > commit 3b2f54bcb437f98e6137c904045cc51072b5c06b > > >> >> > > [hotfix][tests] Move constants in SplitFetcherTest relevant to > > only > > >> >> one > > >> >> > > test into test method > > >> >> > > > > >> >> > > commit d7625760a75a508bf05bcddc380bb4d62ee1743e > > >> >> > > [FLINK-19225][connectors] Various small improvements to > > >> >> SourceReaderBase > > >> >> > > > > >> >> > > commit a5b0d3297748c1be47ad579a88f24df2255a8df1 > > >> >> > > [FLINK-17393][connectors] Wakeup the SplitFetchers more > > elegantly. > > >> >> > > > > >> >> > > commit f42a3ebc3e81a034b7221a803c153636fef34903 > > >> >> > > [FLINK-18680][connectors] Make connector base > RecordsWithSplitIds > > >> more > > >> >> > > lightweight. > > >> >> > > > > >> >> > > commit e3d273de822b085183d09b275a445879ff94b350 > > >> >> > > [FLINK-19162][connectors] Add 'recycle()' to the > > >> RecordsWithSplitIds > > >> >> to > > >> >> > > support reuse of heavy objects. > > >> >> > > > > >> >> > > commit 8ebc464c2520453a70001cd712abc8dee6ee89e0 > > >> >> > > [hotfix][testing] Add a set of parameterizable testing mocks > for > > >> the > > >> >> > Split > > >> >> > > Reader API > > >> >> > > > > >> >> > > commit 930a07438be1185388d7150640f294dfe2a5d378 > > >> >> > > [hotfix][connectors] Add RequestSplitEvent to > > >> 'flink-connector-base' > > >> >> > > > > >> >> > > commit e945ce8a933bc378844782f784ca473c767ca159 > > >> >> > > [refactor][connectors] Backport of the connector-base exception > > >> >> handling > > >> >> > > from the Kafka Connector Pull Request > > >> >> > > > > >> >> > > > > >> >> > > On Wed, Oct 28, 2020 at 5:49 PM Steven Wu < > stevenz...@gmail.com> > > >> >> wrote: > > >> >> > > > > >> >> > > > I would love to see this FLIP-27 source interface improvement > > [1] > > >> >> made > > >> >> > to > > >> >> > > > 1.11.3. > > >> >> > > > > > >> >> > > > [1] https://issues.apache.org/jira/browse/FLINK-19698 > > >> >> > > > > > >> >> > > > On Wed, Oct 28, 2020 at 12:32 AM Tzu-Li (Gordon) Tai < > > >> >> > > tzuli...@apache.org> > > >> >> > > > wrote: > > >> >> > > > > > >> >> > > > > Thanks for the replies so far! > > >> >> > > > > > > >> >> > > > > Just to provide a brief update on the status of blockers > for > > >> >> 1.11.3 > > >> >> > so > > >> >> > > > far: > > >> >> > > > > > > >> >> > > > > > > >> >> > > > > *PR opened, pending reviewer*- [FLINK-19717] > > >> >> > SourceReaderBase.pollNext > > >> >> > > > may > > >> >> > > > > return END_OF_INPUT if SplitReader.fetch throws ( > > >> >> > > > > https://github.com/apache/flink/pull/13776) > > >> >> > > > > > > >> >> > > > > *PR opened, reviewed + close to being merged* > > >> >> > > > > - [FLINK-19741] Timer service should skip restoring from > raw > > >> keyed > > >> >> > > stream > > >> >> > > > > if it isn't the writer ( > > >> >> https://github.com/apache/flink/pull/13761) > > >> >> > > > > - [FLINK-19748] Raw keyed stream key group iterator should > be > > >> >> > skipping > > >> >> > > > > unwritten key groups ( > > >> https://github.com/apache/flink/pull/13772) > > >> >> > > > > > > >> >> > > > > *Merged* > > >> >> > > > > - [FLINK-19154] Application mode deletes HA data in case of > > >> >> suspended > > >> >> > > > > ZooKeeper connection > > >> >> > > > > - [FLINK-19569] Upgrade ICU4J to 67.1+ > > >> >> > > > > > > >> >> > > > > Right now as it seems, progress is mainly blocked on a > > reviewer > > >> >> for > > >> >> > > > > FLINK-19717. > > >> >> > > > > Meanwhile, Xintong is keeping an eye on test instabilities > > [1] > > >> to > > >> >> see > > >> >> > > if > > >> >> > > > > there are any fixes that should be applied to > `release-1.11`. > > >> >> > > > > > > >> >> > > > > This is also a reminder, that if there are other blockers > > that > > >> we > > >> >> > need > > >> >> > > to > > >> >> > > > > be aware of, or a need to re-establish estimated time for > > >> getting > > >> >> > fixes > > >> >> > > > in > > >> >> > > > > and delay the RC for 1.11.3, please do let us know! > > >> >> > > > > > > >> >> > > > > Cheers, > > >> >> > > > > Gordon > > >> >> > > > > > > >> >> > > > > [1] > > >> >> > > > > > > >> >> > > > > > > >> >> > > > > > >> >> > > > > >> >> > > > >> >> > > >> > > > https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20affectedVersion%20in%20(1.11.0%2C%201.11.1%2C%201.11.2)%20AND%20labels%20%3D%20test-stability > > >> >> > > > > > > >> >> > > > > On Mon, Oct 26, 2020 at 9:43 PM Kostas Kloudas < > > >> >> kklou...@apache.org> > > >> >> > > > > wrote: > > >> >> > > > > > > >> >> > > > > > +1 for releasing Flink 1.11.3 as it contains a number of > > >> >> important > > >> >> > > > > > fixes and thanks Gordon and Xintong for volunteering. > > >> >> > > > > > > > >> >> > > > > > Cheers, > > >> >> > > > > > Kostas > > >> >> > > > > > > > >> >> > > > > > On Mon, Oct 26, 2020 at 4:37 AM Yu Li <car...@gmail.com> > > >> wrote: > > >> >> > > > > > > > > >> >> > > > > > > +1 for releasing Flink 1.11.3, and thanks Gordon and > > >> Xintong > > >> >> for > > >> >> > > > > > > volunteering as our release managers. > > >> >> > > > > > > > > >> >> > > > > > > Best Regards, > > >> >> > > > > > > Yu > > >> >> > > > > > > > > >> >> > > > > > > > > >> >> > > > > > > On Mon, 26 Oct 2020 at 09:45, Xintong Song < > > >> >> > tonysong...@gmail.com> > > >> >> > > > > > wrote: > > >> >> > > > > > > > > >> >> > > > > > > > Thanks Gordan for starting this discussion. > > >> >> > > > > > > > My pleasure to help with the release process. > > >> >> > > > > > > > > > >> >> > > > > > > > Thank you~ > > >> >> > > > > > > > > > >> >> > > > > > > > Xintong Song > > >> >> > > > > > > > > > >> >> > > > > > > > > > >> >> > > > > > > > > > >> >> > > > > > > > On Fri, Oct 23, 2020 at 11:29 PM Till Rohrmann < > > >> >> > > > trohrm...@apache.org > > >> >> > > > > > > > >> >> > > > > > > > wrote: > > >> >> > > > > > > > > > >> >> > > > > > > > > Thanks for starting this discussion Gordon. There > are > > >> over > > >> >> > 100 > > >> >> > > > > issues > > >> >> > > > > > > > > which are fixed for 1.11.3. Hence +1 for a soonish > > >> 1.11.3 > > >> >> > > > release. > > >> >> > > > > > Thanks > > >> >> > > > > > > > > for volunteering as our release managers Gordon and > > >> >> Xintong! > > >> >> > > > > > > > > > > >> >> > > > > > > > > Cheers, > > >> >> > > > > > > > > Till > > >> >> > > > > > > > > > > >> >> > > > > > > > > On Fri, Oct 23, 2020 at 5:02 PM Tzu-Li (Gordon) > Tai < > > >> >> > > > > > tzuli...@apache.org > > >> >> > > > > > > > > > > >> >> > > > > > > > > wrote: > > >> >> > > > > > > > > > > >> >> > > > > > > > >> Hi, > > >> >> > > > > > > > >> > > >> >> > > > > > > > >> Xintong and I would like to start a discussion for > > >> >> releasing > > >> >> > > > Flink > > >> >> > > > > > > > 1.11.3 > > >> >> > > > > > > > >> soon. > > >> >> > > > > > > > >> > > >> >> > > > > > > > >> It seems like we already have a few pressing > issues > > >> that > > >> >> > needs > > >> >> > > > to > > >> >> > > > > be > > >> >> > > > > > > > >> included in a new hotfix release: > > >> >> > > > > > > > >> > > >> >> > > > > > > > >> - Heap-based timers’ restore behaviour is > > causing a > > >> >> > > critical > > >> >> > > > > > recovery > > >> >> > > > > > > > >> issue for StateFun [1] [2] [3]. > > >> >> > > > > > > > >> - There are several robustness issues for the > > >> FLIP-27 > > >> >> new > > >> >> > > > > source > > >> >> > > > > > API, > > >> >> > > > > > > > >> such as [4]. We already have some users using > the > > >> >> FLIP-27 > > >> >> > > API > > >> >> > > > > > with > > >> >> > > > > > > > >> 1.11.x, > > >> >> > > > > > > > >> so it would be important to get those fixes in > > for > > >> >> 1.11.x > > >> >> > > as > > >> >> > > > > > well. > > >> >> > > > > > > > >> > > >> >> > > > > > > > >> Apart from the issues that are already marked as > > >> blocker > > >> >> for > > >> >> > > > > 1.11.3 > > >> >> > > > > > in > > >> >> > > > > > > > our > > >> >> > > > > > > > >> JIRA [5], please let us know in this thread if > there > > >> is > > >> >> > > already > > >> >> > > > > > ongoing > > >> >> > > > > > > > >> work for other important fixes that we should try > to > > >> >> > include. > > >> >> > > > > > > > >> > > >> >> > > > > > > > >> Xintong and I would like to volunteer for managing > > >> this > > >> >> > > release, > > >> >> > > > > and > > >> >> > > > > > > > will > > >> >> > > > > > > > >> try to communicate the priority of pending > blockers > > >> over > > >> >> the > > >> >> > > > next > > >> >> > > > > > few > > >> >> > > > > > > > >> days. > > >> >> > > > > > > > >> Since the aforementioned issues are quite > critical, > > >> we’d > > >> >> > like > > >> >> > > to > > >> >> > > > > aim > > >> >> > > > > > > > >> for a *feature > > >> >> > > > > > > > >> freeze by the end of next week (Oct. 30th)* and > > start > > >> the > > >> >> > > > release > > >> >> > > > > > voting > > >> >> > > > > > > > >> process the week after. > > >> >> > > > > > > > >> If that is too short of a notice and you might > need > > >> more > > >> >> > time, > > >> >> > > > > > please > > >> >> > > > > > > > let > > >> >> > > > > > > > >> us know! > > >> >> > > > > > > > >> > > >> >> > > > > > > > >> Cheers, > > >> >> > > > > > > > >> Gordon > > >> >> > > > > > > > >> > > >> >> > > > > > > > >> [1] > > https://issues.apache.org/jira/browse/FLINK-19692 > > >> >> > > > > > > > >> [2] > > https://issues.apache.org/jira/browse/FLINK-19741 > > >> >> > > > > > > > >> [3] > > https://issues.apache.org/jira/browse/FLINK-19748 > > >> >> > > > > > > > >> [4] > > https://issues.apache.org/jira/browse/FLINK-19717 > > >> >> > > > > > > > >> [5] > > >> >> > > > > > > > >> > > >> >> > > > > > > > >> > > >> >> > > > > > > > > > >> >> > > > > > > > >> >> > > > > > > >> >> > > > > > >> >> > > > > >> >> > > > >> >> > > >> > > > https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20AND%20priority%20%3D%20Blocker%20AND%20fixVersion%20%3D%201.11.3 > > >> >> > > > > > > > >> > > >> >> > > > > > > > > > > >> >> > > > > > > > > > >> >> > > > > > > > >> >> > > > > > > >> >> > > > > > >> >> > > > > >> >> > > > >> >> > > > >> >> > -- > > >> >> > Best, Jingsong Lee > > >> >> > > > >> >> > > >> > > > >> > > > > > >
