Hi All, Just an update. All known blockers are resolved and I'm starting to create RC1 for release 1.11.2.
Thanks, Zhu Zhu Zhu <reed...@gmail.com> 于2020年9月9日周三 上午11:36写道： > Thanks for reporting this issue and offering to fix it @Jingsong Li > <jingsongl...@gmail.com> > Agreed it is a reasonable blocker. I will postpone 1.11.2 RC1 creation > until it is fixed. > > Thanks, > Zhu > > Jingsong Li <jingsongl...@gmail.com> 于2020年9月9日周三 上午11:27写道： > >> Hi Zhu Zhu, >> >> Replenish its[1] influence: >> For HiveStreamingSink & FileSystemSink in Table/SQL, partition commit make >> partition visible for downstream Hive/Spark engines. >> But due to FLINK-19166, will lose some partitions to commit after Job >> failover in some cases, especially for short partitions. >> In the user's opinion, the data is lost, which does not conform to >> exactly-once. >> >> [1]https://issues.apache.org/jira/browse/FLINK-19166 >> [2] >> >> https://ci.apache.org/projects/flink/flink-docs-release-1.11/dev/table/connectors/filesystem.html#partition-commit >> >> Best, >> Jingsong >> >> On Wed, Sep 9, 2020 at 11:08 AM Jingsong Li <jingsongl...@gmail.com> >> wrote: >> >> > Hi Zhu Zhu, >> > >> > Add a new blocker: https://issues.apache.org/jira/browse/FLINK-19166 >> > >> > Will fix it soon. >> > >> > Best, >> > Jingsong >> > >> > On Tue, Sep 8, 2020 at 12:26 AM Zhu Zhu <reed...@gmail.com> wrote: >> > >> >> Hi All, >> >> >> >> Since there are still two 1.11.2 blockers items in progress, RC1 >> creation >> >> will be postponed to tomorrow. >> >> >> >> Thanks, >> >> Zhu >> >> >> >> Zhu Zhu <reed...@gmail.com> 于2020年9月4日周五 下午3:50写道： >> >> >> >> > @Dawid >> >> > Thanks for the information and preparing a fix for FLINK-19133! >> >> > I have made it a blocker for 1.11.2 and will keep tracking its >> status. >> >> > >> >> > @Till >> >> > Thanks for the updates and efforts for FLINK-18959! >> >> > >> >> > Thanks, >> >> > Zhu >> >> > >> >> > Till Rohrmann <trohrm...@apache.org> 于2020年9月4日周五 下午3:41写道： >> >> > >> >> >> Fyi, FLINK-18959 has been merged into the release-1.10 branch. >> >> >> >> >> >> Cheers, >> >> >> Till >> >> >> >> >> >> On Thu, Sep 3, 2020 at 2:38 PM Dawid Wysakowicz < >> >> dwysakow...@apache.org> >> >> >> wrote: >> >> >> >> >> >>> User has just reported another issue FLINK-19133 which I think >> should >> >> be >> >> >>> a blocker for the 1.11.2 release. I'll try to prepare a fix as >> soon as >> >> >>> possible. >> >> >>> >> >> >>> On 03/09/2020 09:36, Zhu Zhu wrote: >> >> >>> > Thanks for the inputs! >> >> >>> > I have made FLINK-14942 and FLINK-18641 blockers for 1.11.2. >> >> >>> > >> >> >>> > And thanks a lot for offering help, zhijiang! >> >> >>> > >> >> >>> > Thanks, >> >> >>> > Zhu >> >> >>> > >> >> >>> > Congxian Qiu <qcx978132...@gmail.com> 于2020年9月3日周四 下午3:18写道： >> >> >>> > >> >> >>> >> Hi >> >> >>> >> I'd like to include FLINK-14942 into 1.11.2. FLINK-14942 >> (this >> >> >>> fixes a >> >> >>> >> bug introduce in 1.11.0), there is a pr for it already. >> >> >>> >> Best, >> >> >>> >> Congxian >> >> >>> >> >> >> >>> >> >> >> >>> >> Zhou, Brian <b.z...@dell.com> 于2020年9月3日周四 上午11:21写道： >> >> >>> >> >> >> >>> >>> Hi, >> >> >>> >>> >> >> >>> >>> Thanks Becket for addressing the issue. FLINK-18641 is now a >> >> blocker >> >> >>> for >> >> >>> >>> Pravega connector integration, hope we can have it included in >> >> 1.11.2 >> >> >>> >>> release. >> >> >>> >>> >> >> >>> >>> Best Regards, >> >> >>> >>> Brian >> >> >>> >>> >> >> >>> >>> -----Original Message----- >> >> >>> >>> From: Becket Qin <becket....@gmail.com> >> >> >>> >>> Sent: Thursday, September 3, 2020 11:18 >> >> >>> >>> To: dev >> >> >>> >>> Cc: khachatryan.ro...@gmail.com; Till Rohrmann; david; >> Jingsong >> >> Li >> >> >>> >>> Subject: Re: [DISCUSS] Releasing Flink 1.11.2 >> >> >>> >>> >> >> >>> >>> >> >> >>> >>> [EXTERNAL EMAIL] >> >> >>> >>> >> >> >>> >>> Hi Zhuzhu, >> >> >>> >>> >> >> >>> >>> Thanks for starting the discussion. >> >> >>> >>> >> >> >>> >>> I'd like to include FLINK-18641 into 1.11.2 as well. It is a >> >> >>> regression >> >> >>> >>> from previous versions and is currently blocking the >> development >> >> of >> >> >>> >> Pravega >> >> >>> >>> source on top of FLIP-27. >> >> >>> >>> >> >> >>> >>> Thanks, >> >> >>> >>> >> >> >>> >>> Jiangjie (Becket) Qin >> >> >>> >>> >> >> >>> >>> On Wed, Sep 2, 2020 at 11:13 PM Zhu Zhu <reed...@gmail.com> >> >> wrote: >> >> >>> >>> >> >> >>> >>>> Thank you all for the inputs! >> >> >>> >>>> >> >> >>> >>>> I agree with Till that we should set a soft deadline first. >> >> >>> >>>> I'd like to propose next Monday if no new blocker issue pops >> up. >> >> >>> >>>> But feel free to raise your concerns if you feel next Monday >> as a >> >> >>> >>>> deadline may not work for fixes which should be a blocker for >> >> >>> 1.11.2. >> >> >>> >>>> >> >> >>> >>>> Here's a summary of the wanted/blocker but still open fixes: >> >> >>> >>>> - FLINK-19121 Avoid accessing HDFS frequently in >> >> >>> HiveBulkWriterFactory >> >> >>> >>>> - FLINK-19109 Split Reader eats chained periodic watermarks >> >> >>> >>>> - (not a strict blocker) FLINK-18959 Fail to >> >> archiveExecutionGraph >> >> >>> >>>> because job is not finished when dispatcher close >> >> >>> >>>> - FLINK-18934 Idle stream does not advance watermark in >> connected >> >> >>> >>>> stream >> >> >>> >>>> >> >> >>> >>>> Thanks, >> >> >>> >>>> Zhu >> >> >>> >>>> >> >> >>> >>>> Konstantin Knauf <kna...@apache.org> 于2020年9月2日周三 下午9:00写道： >> >> >>> >>>> >> >> >>> >>>>> I think it would be nice to include a fix for >> >> >>> >>>>> https://issues.apache.org/jira/browse/FLINK-18934, too, as >> it >> >> >>> >>>>> affects a highly requested feature of Flink 1.11 quite >> severely. >> >> >>> >>>>> >> >> >>> >>>>> On Wed, Sep 2, 2020 at 2:51 PM Till Rohrmann < >> >> trohrm...@apache.org >> >> >>> > >> >> >>> >>>> wrote: >> >> >>> >>>>>> Thanks a lot for starting this discussion Zhu Zhu and for >> >> >>> >>>>>> volunteering >> >> >>> >>>> as >> >> >>> >>>>>> the release manager. Big +1 for creating the next 1.11 bug >> fix >> >> >>> >>>> release. I >> >> >>> >>>>>> think we already collected quite a bit of fixes which our >> users >> >> >>> >>>>>> will benefit from. >> >> >>> >>>>>> >> >> >>> >>>>>> For the pending fixes, I would suggest setting a soft >> deadline >> >> >>> >>>>>> (maybe >> >> >>> >>>>> until >> >> >>> >>>>>> beginning of next week) and then starting to cut the release >> >> >>> >>>>>> (given >> >> >>> >>>> that >> >> >>> >>>>> no >> >> >>> >>>>>> other blocker issues pop up). Maybe we are able to resolve >> the >> >> >>> >>>>>> issues >> >> >>> >>>>> even >> >> >>> >>>>>> earlier which would allow us to cut the release also >> earlier. >> >> >>> >>>>>> >> >> >>> >>>>>> From my side I would like to include FLINK-18959 in the >> >> release. >> >> >>> >>>>>> But it >> >> >>> >>>>> is >> >> >>> >>>>>> not a strict release blocker. >> >> >>> >>>>>> >> >> >>> >>>>>> Cheers, >> >> >>> >>>>>> Till >> >> >>> >>>>>> >> >> >>> >>>>>> On Wed, Sep 2, 2020 at 2:40 PM David Anderson >> >> >>> >>>>>> <da...@alpinegizmo.com> >> >> >>> >>>>>> wrote: >> >> >>> >>>>>> >> >> >>> >>>>>>> I think it's worth considering whether we can get this >> bugfix >> >> >>> >>>> included >> >> >>> >>>>> in >> >> >>> >>>>>>> 1.11.2: >> >> >>> >>>>>>> >> >> >>> >>>>>>> - FLINK-19109 Split Reader eats chained periodic watermarks >> >> >>> >>>>>>> >> >> >>> >>>>>>> There is a PR, but it's still a work in progress. Cc'ing >> >> Roman, >> >> >>> >>>>>>> who >> >> >>> >>>> has >> >> >>> >>>>>>> been working on this. >> >> >>> >>>>>>> >> >> >>> >>>>>>> Regards, >> >> >>> >>>>>>> David >> >> >>> >>>>>>> >> >> >>> >>>>>>> >> >> >>> >>>>>>> On Wed, Sep 2, 2020 at 2:19 PM Zhu Zhu <reed...@gmail.com> >> >> >>> >> wrote: >> >> >>> >>>>>>>> Hi All, >> >> >>> >>>>>>>> >> >> >>> >>>>>>>> It has been about 1 month since we released Flink 1.11.1. >> >> It's >> >> >>> >>>>>>>> not >> >> >>> >>>>> too >> >> >>> >>>>>>> far >> >> >>> >>>>>>>> but >> >> >>> >>>>>>>> we already have more than 80 resolved improvements/bugs in >> >> the >> >> >>> >>>>>>> release-1.11 >> >> >>> >>>>>>>> branch. Some of them are quite critical. Therefore, I >> propose >> >> >>> >>>>>>>> to >> >> >>> >>>>> create >> >> >>> >>>>>>> the >> >> >>> >>>>>>>> next >> >> >>> >>>>>>>> bugfix release 1.11.2 for Flink 1.11. >> >> >>> >>>>>>>> >> >> >>> >>>>>>>> Most noticeable fixes are: >> >> >>> >>>>>>>> - FLINK-18769 MiniBatch doesn't work with FLIP-95 source >> >> >>> >>>>>>>> - FLINK-18902 Cannot serve results of asynchronous REST >> >> >>> >>>>>>>> operations >> >> >>> >>>> in >> >> >>> >>>>>>>> per-job mode >> >> >>> >>>>>>>> - FLINK-18682 Vector orc reader cannot read Hive 2.0.0 >> table >> >> >>> >>>>>>>> - FLINK-18608 CustomizedConvertRule#convertCast drops >> >> >>> >>>>>>>> nullability >> >> >>> >>>>>>>> - FLINK-18646 Managed memory released check can block RPC >> >> >>> >>>>>>>> thread >> >> >>> >>>>>>>> - FLINK-18993 Invoke sanityCheckTotalFlinkMemory method >> >> >>> >>>>>>>> incorrectly >> >> >>> >>>>> in >> >> >>> >>>>>>>> JobManagerFlinkMemoryUtils.java >> >> >>> >>>>>>>> - FLINK-18663 RestServerEndpoint may prevent server >> shutdown >> >> >>> >>>>>>>> - FLINK-18595 Deadlock during job shutdown >> >> >>> >>>>>>>> - FLINK-18581 Cannot find GC cleaner with java version >> >> >>> >>>>>>>> previous >> >> >>> >>>>>>>> jdk8u72(-b01) >> >> >>> >>>>>>>> - FLINK-17075 Add task status reconciliation between TM >> and >> >> JM >> >> >>> >>>>>>>> >> >> >>> >>>>>>>> Furthermore, I think the following blocker issue should be >> >> >>> >>>>>>>> merged >> >> >>> >>>>>> before >> >> >>> >>>>>>>> 1.11.2 release >> >> >>> >>>>>>>> >> >> >>> >>>>>>>> - FLINK-19121 Avoid accessing HDFS frequently in >> >> >>> >>>>> HiveBulkWriterFactory >> >> >>> >>>>>>>> I would volunteer as the release manager and kick off the >> >> >>> >>>>>>>> release >> >> >>> >>>>>>> process. >> >> >>> >>>>>>>> What do you think? >> >> >>> >>>>>>>> >> >> >>> >>>>>>>> Please let me know if there are any concerns or any >> missing >> >> >>> >>>>>>>> blocker >> >> >>> >>>>>>> issues >> >> >>> >>>>>>>> need to be fixed in 1.11.2. >> >> >>> >>>>>>>> >> >> >>> >>>>>>>> Thanks, >> >> >>> >>>>>>>> Zhu Zhu >> >> >>> >>>>>>>> >> >> >>> >>>>> >> >> >>> >>>>> -- >> >> >>> >>>>> >> >> >>> >>>>> Konstantin Knauf >> >> >>> >>>>> >> >> >>> >>>>> https://twitter.com/snntrable >> >> >>> >>>>> >> >> >>> >>>>> https://github.com/knaufk >> >> >>> >>>>> >> >> >>> >> >> >>> >> >> >> > >> > >> > -- >> > Best, Jingsong Lee >> > >> >> >> -- >> Best, Jingsong Lee >> >
