Thanks for reporting this issue and offering to fix it @Jingsong Li <jingsongl...@gmail.com> Agreed it is a reasonable blocker. I will postpone 1.11.2 RC1 creation until it is fixed.
Thanks, Zhu Jingsong Li <jingsongl...@gmail.com> 于2020年9月9日周三 上午11:27写道： > Hi Zhu Zhu, > > Replenish its[1] influence: > For HiveStreamingSink & FileSystemSink in Table/SQL, partition commit make > partition visible for downstream Hive/Spark engines. > But due to FLINK-19166, will lose some partitions to commit after Job > failover in some cases, especially for short partitions. > In the user's opinion, the data is lost, which does not conform to > exactly-once. > > [1]https://issues.apache.org/jira/browse/FLINK-19166 > [2] > > https://ci.apache.org/projects/flink/flink-docs-release-1.11/dev/table/connectors/filesystem.html#partition-commit > > Best, > Jingsong > > On Wed, Sep 9, 2020 at 11:08 AM Jingsong Li <jingsongl...@gmail.com> > wrote: > > > Hi Zhu Zhu, > > > > Add a new blocker: https://issues.apache.org/jira/browse/FLINK-19166 > > > > Will fix it soon. > > > > Best, > > Jingsong > > > > On Tue, Sep 8, 2020 at 12:26 AM Zhu Zhu <reed...@gmail.com> wrote: > > > >> Hi All, > >> > >> Since there are still two 1.11.2 blockers items in progress, RC1 > creation > >> will be postponed to tomorrow. > >> > >> Thanks, > >> Zhu > >> > >> Zhu Zhu <reed...@gmail.com> 于2020年9月4日周五 下午3:50写道： > >> > >> > @Dawid > >> > Thanks for the information and preparing a fix for FLINK-19133! > >> > I have made it a blocker for 1.11.2 and will keep tracking its status. > >> > > >> > @Till > >> > Thanks for the updates and efforts for FLINK-18959! > >> > > >> > Thanks, > >> > Zhu > >> > > >> > Till Rohrmann <trohrm...@apache.org> 于2020年9月4日周五 下午3:41写道： > >> > > >> >> Fyi, FLINK-18959 has been merged into the release-1.10 branch. > >> >> > >> >> Cheers, > >> >> Till > >> >> > >> >> On Thu, Sep 3, 2020 at 2:38 PM Dawid Wysakowicz < > >> dwysakow...@apache.org> > >> >> wrote: > >> >> > >> >>> User has just reported another issue FLINK-19133 which I think > should > >> be > >> >>> a blocker for the 1.11.2 release. I'll try to prepare a fix as soon > as > >> >>> possible. > >> >>> > >> >>> On 03/09/2020 09:36, Zhu Zhu wrote: > >> >>> > Thanks for the inputs! > >> >>> > I have made FLINK-14942 and FLINK-18641 blockers for 1.11.2. > >> >>> > > >> >>> > And thanks a lot for offering help, zhijiang! > >> >>> > > >> >>> > Thanks, > >> >>> > Zhu > >> >>> > > >> >>> > Congxian Qiu <qcx978132...@gmail.com> 于2020年9月3日周四 下午3:18写道： > >> >>> > > >> >>> >> Hi > >> >>> >> I'd like to include FLINK-14942 into 1.11.2. FLINK-14942 > (this > >> >>> fixes a > >> >>> >> bug introduce in 1.11.0), there is a pr for it already. > >> >>> >> Best, > >> >>> >> Congxian > >> >>> >> > >> >>> >> > >> >>> >> Zhou, Brian <b.z...@dell.com> 于2020年9月3日周四 上午11:21写道： > >> >>> >> > >> >>> >>> Hi, > >> >>> >>> > >> >>> >>> Thanks Becket for addressing the issue. FLINK-18641 is now a > >> blocker > >> >>> for > >> >>> >>> Pravega connector integration, hope we can have it included in > >> 1.11.2 > >> >>> >>> release. > >> >>> >>> > >> >>> >>> Best Regards, > >> >>> >>> Brian > >> >>> >>> > >> >>> >>> -----Original Message----- > >> >>> >>> From: Becket Qin <becket....@gmail.com> > >> >>> >>> Sent: Thursday, September 3, 2020 11:18 > >> >>> >>> To: dev > >> >>> >>> Cc: khachatryan.ro...@gmail.com; Till Rohrmann; david; Jingsong > >> Li > >> >>> >>> Subject: Re: [DISCUSS] Releasing Flink 1.11.2 > >> >>> >>> > >> >>> >>> > >> >>> >>> [EXTERNAL EMAIL] > >> >>> >>> > >> >>> >>> Hi Zhuzhu, > >> >>> >>> > >> >>> >>> Thanks for starting the discussion. > >> >>> >>> > >> >>> >>> I'd like to include FLINK-18641 into 1.11.2 as well. It is a > >> >>> regression > >> >>> >>> from previous versions and is currently blocking the development > >> of > >> >>> >> Pravega > >> >>> >>> source on top of FLIP-27. > >> >>> >>> > >> >>> >>> Thanks, > >> >>> >>> > >> >>> >>> Jiangjie (Becket) Qin > >> >>> >>> > >> >>> >>> On Wed, Sep 2, 2020 at 11:13 PM Zhu Zhu <reed...@gmail.com> > >> wrote: > >> >>> >>> > >> >>> >>>> Thank you all for the inputs! > >> >>> >>>> > >> >>> >>>> I agree with Till that we should set a soft deadline first. > >> >>> >>>> I'd like to propose next Monday if no new blocker issue pops > up. > >> >>> >>>> But feel free to raise your concerns if you feel next Monday > as a > >> >>> >>>> deadline may not work for fixes which should be a blocker for > >> >>> 1.11.2. > >> >>> >>>> > >> >>> >>>> Here's a summary of the wanted/blocker but still open fixes: > >> >>> >>>> - FLINK-19121 Avoid accessing HDFS frequently in > >> >>> HiveBulkWriterFactory > >> >>> >>>> - FLINK-19109 Split Reader eats chained periodic watermarks > >> >>> >>>> - (not a strict blocker) FLINK-18959 Fail to > >> archiveExecutionGraph > >> >>> >>>> because job is not finished when dispatcher close > >> >>> >>>> - FLINK-18934 Idle stream does not advance watermark in > connected > >> >>> >>>> stream > >> >>> >>>> > >> >>> >>>> Thanks, > >> >>> >>>> Zhu > >> >>> >>>> > >> >>> >>>> Konstantin Knauf <kna...@apache.org> 于2020年9月2日周三 下午9:00写道： > >> >>> >>>> > >> >>> >>>>> I think it would be nice to include a fix for > >> >>> >>>>> https://issues.apache.org/jira/browse/FLINK-18934, too, as it > >> >>> >>>>> affects a highly requested feature of Flink 1.11 quite > severely. > >> >>> >>>>> > >> >>> >>>>> On Wed, Sep 2, 2020 at 2:51 PM Till Rohrmann < > >> trohrm...@apache.org > >> >>> > > >> >>> >>>> wrote: > >> >>> >>>>>> Thanks a lot for starting this discussion Zhu Zhu and for > >> >>> >>>>>> volunteering > >> >>> >>>> as > >> >>> >>>>>> the release manager. Big +1 for creating the next 1.11 bug > fix > >> >>> >>>> release. I > >> >>> >>>>>> think we already collected quite a bit of fixes which our > users > >> >>> >>>>>> will benefit from. > >> >>> >>>>>> > >> >>> >>>>>> For the pending fixes, I would suggest setting a soft > deadline > >> >>> >>>>>> (maybe > >> >>> >>>>> until > >> >>> >>>>>> beginning of next week) and then starting to cut the release > >> >>> >>>>>> (given > >> >>> >>>> that > >> >>> >>>>> no > >> >>> >>>>>> other blocker issues pop up). Maybe we are able to resolve > the > >> >>> >>>>>> issues > >> >>> >>>>> even > >> >>> >>>>>> earlier which would allow us to cut the release also earlier. > >> >>> >>>>>> > >> >>> >>>>>> From my side I would like to include FLINK-18959 in the > >> release. > >> >>> >>>>>> But it > >> >>> >>>>> is > >> >>> >>>>>> not a strict release blocker. > >> >>> >>>>>> > >> >>> >>>>>> Cheers, > >> >>> >>>>>> Till > >> >>> >>>>>> > >> >>> >>>>>> On Wed, Sep 2, 2020 at 2:40 PM David Anderson > >> >>> >>>>>> <da...@alpinegizmo.com> > >> >>> >>>>>> wrote: > >> >>> >>>>>> > >> >>> >>>>>>> I think it's worth considering whether we can get this > bugfix > >> >>> >>>> included > >> >>> >>>>> in > >> >>> >>>>>>> 1.11.2: > >> >>> >>>>>>> > >> >>> >>>>>>> - FLINK-19109 Split Reader eats chained periodic watermarks > >> >>> >>>>>>> > >> >>> >>>>>>> There is a PR, but it's still a work in progress. Cc'ing > >> Roman, > >> >>> >>>>>>> who > >> >>> >>>> has > >> >>> >>>>>>> been working on this. > >> >>> >>>>>>> > >> >>> >>>>>>> Regards, > >> >>> >>>>>>> David > >> >>> >>>>>>> > >> >>> >>>>>>> > >> >>> >>>>>>> On Wed, Sep 2, 2020 at 2:19 PM Zhu Zhu <reed...@gmail.com> > >> >>> >> wrote: > >> >>> >>>>>>>> Hi All, > >> >>> >>>>>>>> > >> >>> >>>>>>>> It has been about 1 month since we released Flink 1.11.1. > >> It's > >> >>> >>>>>>>> not > >> >>> >>>>> too > >> >>> >>>>>>> far > >> >>> >>>>>>>> but > >> >>> >>>>>>>> we already have more than 80 resolved improvements/bugs in > >> the > >> >>> >>>>>>> release-1.11 > >> >>> >>>>>>>> branch. Some of them are quite critical. Therefore, I > propose > >> >>> >>>>>>>> to > >> >>> >>>>> create > >> >>> >>>>>>> the > >> >>> >>>>>>>> next > >> >>> >>>>>>>> bugfix release 1.11.2 for Flink 1.11. > >> >>> >>>>>>>> > >> >>> >>>>>>>> Most noticeable fixes are: > >> >>> >>>>>>>> - FLINK-18769 MiniBatch doesn't work with FLIP-95 source > >> >>> >>>>>>>> - FLINK-18902 Cannot serve results of asynchronous REST > >> >>> >>>>>>>> operations > >> >>> >>>> in > >> >>> >>>>>>>> per-job mode > >> >>> >>>>>>>> - FLINK-18682 Vector orc reader cannot read Hive 2.0.0 > table > >> >>> >>>>>>>> - FLINK-18608 CustomizedConvertRule#convertCast drops > >> >>> >>>>>>>> nullability > >> >>> >>>>>>>> - FLINK-18646 Managed memory released check can block RPC > >> >>> >>>>>>>> thread > >> >>> >>>>>>>> - FLINK-18993 Invoke sanityCheckTotalFlinkMemory method > >> >>> >>>>>>>> incorrectly > >> >>> >>>>> in > >> >>> >>>>>>>> JobManagerFlinkMemoryUtils.java > >> >>> >>>>>>>> - FLINK-18663 RestServerEndpoint may prevent server > shutdown > >> >>> >>>>>>>> - FLINK-18595 Deadlock during job shutdown > >> >>> >>>>>>>> - FLINK-18581 Cannot find GC cleaner with java version > >> >>> >>>>>>>> previous > >> >>> >>>>>>>> jdk8u72(-b01) > >> >>> >>>>>>>> - FLINK-17075 Add task status reconciliation between TM and > >> JM > >> >>> >>>>>>>> > >> >>> >>>>>>>> Furthermore, I think the following blocker issue should be > >> >>> >>>>>>>> merged > >> >>> >>>>>> before > >> >>> >>>>>>>> 1.11.2 release > >> >>> >>>>>>>> > >> >>> >>>>>>>> - FLINK-19121 Avoid accessing HDFS frequently in > >> >>> >>>>> HiveBulkWriterFactory > >> >>> >>>>>>>> I would volunteer as the release manager and kick off the > >> >>> >>>>>>>> release > >> >>> >>>>>>> process. > >> >>> >>>>>>>> What do you think? > >> >>> >>>>>>>> > >> >>> >>>>>>>> Please let me know if there are any concerns or any missing > >> >>> >>>>>>>> blocker > >> >>> >>>>>>> issues > >> >>> >>>>>>>> need to be fixed in 1.11.2. > >> >>> >>>>>>>> > >> >>> >>>>>>>> Thanks, > >> >>> >>>>>>>> Zhu Zhu > >> >>> >>>>>>>> > >> >>> >>>>> > >> >>> >>>>> -- > >> >>> >>>>> > >> >>> >>>>> Konstantin Knauf > >> >>> >>>>> > >> >>> >>>>> https://twitter.com/snntrable > >> >>> >>>>> > >> >>> >>>>> https://github.com/knaufk > >> >>> >>>>> > >> >>> > >> >>> > >> > > > > > > -- > > Best, Jingsong Lee > > > > > -- > Best, Jingsong Lee >
