Hi Jincheng, Thanks a lot for your timely help. I'm on my way to the release.
Best, Hequn On Wed, Nov 27, 2019 at 7:36 AM jincheng sun <sunjincheng...@gmail.com> wrote: > Hi Hequn, > > Thank you for your great job! Looking forward the first RC of 1.8.3 ! > BTW: The version of 1.8.4 already created here: > https://issues.apache.org/jira/projects/FLINK/versions/12346552 > > Best, > Jincheng > > Hequn Cheng <chenghe...@gmail.com> 于2019年11月26日周二 下午8:18写道： > >> Hi all, >> >> I would like to share with you that all blockers are resolved now. If >> there are no more critical issues, I will create the first RC tomorrow and >> vote on it directly. >> Hope everything goes well! >> >> Thank you all for the help of fixing, reviewing, driving and discussions! >> >> Best, Hequn >> >> On Tue, Nov 26, 2019 at 9:27 AM Hequn Cheng <chenghe...@gmail.com> wrote: >> >>> Hi, >>> >>> @Ufuk Celebi <u...@apache.org> Hi, we are very close now. There is one >>> issue(FLINK-13995 <https://issues.apache.org/jira/browse/FLINK-13995>) >>> left that I want to double-check with you guys. Once this is done, we can >>> create the first RC. I already have some minor comments in the PR >>> <https://github.com/apache/flink/pull/10195>. >>> >>> @Zhu Zhu <reed...@gmail.com> Glad to hear that it is not a blocker. >>> Thank you. >>> >>> Best, Hequn >>> >>> On Mon, Nov 25, 2019 at 5:43 PM Ufuk Celebi <u...@apache.org> wrote: >>> >>>> @Hequn: flink-shaded:9.0 is available in Maven central now. I think you >>>> can go ahead and create the first RC. :-) >>>> >>>> On Mon, Nov 25, 2019 at 7:47 AM Zhu Zhu <reed...@gmail.com> wrote: >>>> >>>>> Hi Hequn, >>>>> >>>>> Looks we are not able to merge fix of FLINK-14735 to 1.8 very soon. >>>>> Given that this fix is for batch job only and batch is not very good in >>>>> 1.8, I think it is a not blocker of release 1.8.3. >>>>> So just don't be blocked by it and feel free to cut the RC when other >>>>> blocking issues are resolved. >>>>> >>>>> Thanks, >>>>> Zhu Zhu >>>>> >>>>> Hequn Cheng <chenghe...@gmail.com> 于2019年11月23日周六 下午9:08写道： >>>>> >>>>> > Hi Zhu Zhu, >>>>> > >>>>> > Thanks a lot for letting us know! >>>>> > We can't cut the first RC right now due to the wait of the >>>>> flink-shade >>>>> > release, so go ahead. >>>>> > >>>>> > Theoretically, we will cut the first RC of 1.8.3 and vote for it >>>>> once the >>>>> > release of flink-shade is done, >>>>> > but I will try my best to have it in 1.8.3. Hope we can get it on >>>>> board on >>>>> > time. :) >>>>> > >>>>> > Best, Hequn >>>>> > >>>>> > On Sat, Nov 23, 2019 at 10:40 AM Zhu Zhu <reed...@gmail.com> wrote: >>>>> > >>>>> >> Hi Jincheng & Hequn >>>>> >> >>>>> >> Thanks for driving the releasing of 1.8.3. >>>>> >> >>>>> >> I am now working on FLINK-14735. The fix avoids duplicated input >>>>> >> checking when scheduling ALL-to-ALL >>>>> >> connected downstream consumers with ALL input constraints. The >>>>> duplicated >>>>> >> checking can cause severe >>>>> >> performance issues for large scale jobs. So I hope the fix could be >>>>> >> released with 1.8.3. >>>>> >> >>>>> >> The fix is already merged into master, and is now in the process of >>>>> >> backporting to 1.8. >>>>> >> >>>>> >> Thanks, >>>>> >> Zhu Zhu >>>>> >> >>>>> >> Ufuk Celebi <u...@apache.org> 于2019年11月15日周五 下午11:54写道： >>>>> >> >>>>> >>> Thanks Chesnay. >>>>> >>> >>>>> >>> I'm also +1 to release 1.8.3 asap without the changes for the >>>>> Jackson >>>>> >>> version bump and leave those for a future release. Realistically, >>>>> the >>>>> >>> flink-shaded release will take until mid next week or end of next >>>>> week. >>>>> >>> But >>>>> >>> please correct me if you think that it should not take that long >>>>> or it's >>>>> >>> OK >>>>> >>> to block the 1.8.3 release on the flink-shaded release. >>>>> >>> >>>>> >>> – Ufuk >>>>> >>> >>>>> >>> >>>>> >>> On Fri, Nov 15, 2019 at 2:27 PM Chesnay Schepler < >>>>> ches...@apache.org> >>>>> >>> wrote: >>>>> >>> >>>>> >>> > I've kicked off a discussion about the next flink-shaded >>>>> release, and >>>>> >>> > have opened PRs for adding the opt-in profile to 1.8/1.9. >>>>> >>> > >>>>> >>> > On 15/11/2019 13:54, Hequn Cheng wrote: >>>>> >>> > > That's great, thank you very much! Ideally, we can kick off the >>>>> >>> release >>>>> >>> > > vote for the first RC of 1.8.3 within next week. :) >>>>> >>> > > >>>>> >>> > > On Fri, Nov 15, 2019 at 8:47 PM Chesnay Schepler < >>>>> ches...@apache.org >>>>> >>> > >>>>> >>> > wrote: >>>>> >>> > > >>>>> >>> > >> I'm not aware of any more planned changes to flink-shaded; so >>>>> we >>>>> >>> could >>>>> >>> > >> start the release right away. >>>>> >>> > >> >>>>> >>> > >> On 15/11/2019 13:44, Hequn Cheng wrote: >>>>> >>> > >>> Hi, >>>>> >>> > >>> >>>>> >>> > >>> @Chesnay Thanks a lot for the explanation. +1 to the opt-in >>>>> >>> approach >>>>> >>> > for >>>>> >>> > >>> 1.8/1.9. >>>>> >>> > >>> @Ufuk Thank you for the nice summary. >>>>> >>> > >>> >>>>> >>> > >>> Looks good so far except that we need to postpone 1.8.3 a >>>>> bit to >>>>> >>> first >>>>> >>> > >> do a >>>>> >>> > >>> flink-shaded release. >>>>> >>> > >>> BTW, @chesnay when would we plan to release the flink-shaded >>>>> with >>>>> >>> > >> upgraded >>>>> >>> > >>> Jackson? >>>>> >>> > >>> >>>>> >>> > >>> Best, Hequn >>>>> >>> > >>> >>>>> >>> > >>> On Fri, Nov 15, 2019 at 7:43 PM Chesnay Schepler < >>>>> >>> ches...@apache.org> >>>>> >>> > >> wrote: >>>>> >>> > >>>> One small modification: the flink-shaded upgrade does not >>>>> have to >>>>> >>> be >>>>> >>> > >>>> part of the profile; since it is only intended for internal >>>>> use >>>>> >>> anyway >>>>> >>> > >>>> (and thus has limited exposure) we can be pretty sure this >>>>> doesn't >>>>> >>> > break >>>>> >>> > >>>> anything. >>>>> >>> > >>>> >>>>> >>> > >>>> On 15/11/2019 12:23, Chesnay Schepler wrote: >>>>> >>> > >>>>> Ufuk's summary is correct. >>>>> >>> > >>>>> >>>>> >>> > >>>>> There's a slight caveat in that we'd also have to bump the >>>>> >>> > >>>>> shade-plugin to 3.1.1 since it otherwise fails on jackson, >>>>> >>> > >>>>> but I have no concerns about this change. >>>>> >>> > >>>>> >>>>> >>> > >>>>> On 15/11/2019 12:19, Ufuk Celebi wrote: >>>>> >>> > >>>>>> The opt-in approach seems reasonable to me. +1 to include >>>>> the >>>>> >>> > >>>>>> profiles in >>>>> >>> > >>>>>> 1.8 and 1.9 without changing the default versions >>>>> (including the >>>>> >>> > >> default >>>>> >>> > >>>>>> version of flink-shaded). >>>>> >>> > >>>>>> >>>>> >>> > >>>>>> As far as I can tell, the next steps would be: >>>>> >>> > >>>>>> >>>>> >>> > >>>>>> 1) Release flink-shaded with upgraded Jackson >>>>> >>> > >>>>>> 2a) Bump the flink-shaded version by default in master >>>>> >>> > >>>>>> 2b) Create opt-in profiles for 1.8 and 1.9 (the opt-in >>>>> profiles >>>>> >>> > >>>>>> should also >>>>> >>> > >>>>>> cover the upgrade to the most recent flink-shaded version) >>>>> >>> > >>>>>> >>>>> >>> > >>>>>> @Chesnay: is this a correct summary? >>>>> >>> > >>>>>> >>>>> >>> > >>>>>> Note this would block the 1.8.3 release on step 1. As an >>>>> >>> upside, we >>>>> >>> > >>>>>> might >>>>> >>> > >>>>>> get some additional feedback until the 1.10 release with >>>>> these >>>>> >>> > >>>>>> profiles in >>>>> >>> > >>>>>> case users make use of them with 1.8/1.9. >>>>> >>> > >>>>>> >>>>> >>> > >>>>>> – Ufuk >>>>> >>> > >>>>>> >>>>> >>> > >>>>>> On Fri, Nov 15, 2019 at 12:08 PM Chesnay Schepler < >>>>> >>> > ches...@apache.org >>>>> >>> > >>>>>> wrote: >>>>> >>> > >>>>>>> The opt-in approach would only be used for 1.8.3 / >>>>> 1.9.2; on >>>>> >>> master >>>>> >>> > >>>>>>> (and >>>>> >>> > >>>>>>> thus starting from 1.10.0) it's not opt-in. >>>>> >>> > >>>>>>> >>>>> >>> > >>>>>>> I have only proposed it as an opt-in because a) we >>>>> usually do >>>>> >>> not >>>>> >>> > >> bump >>>>> >>> > >>>>>>> dependencies in bugfix releases and b) it's a short-term >>>>> change >>>>> >>> > that >>>>> >>> > >> we >>>>> >>> > >>>>>>> aren't allowing to mature properly. >>>>> >>> > >>>>>>> In contrast, the 1.10 release is significantly further >>>>> away, >>>>> >>> hence >>>>> >>> > no >>>>> >>> > >>>>>>> opt-in. >>>>> >>> > >>>>>>> >>>>> >>> > >>>>>>> Hence, I'm not concerned about such kind of ugprades >>>>> being more >>>>> >>> > >> common >>>>> >>> > >>>>>>> in the future. >>>>> >>> > >>>>>>> >>>>> >>> > >>>>>>> We can certainly support every jackson version that >>>>> fixes these >>>>> >>> > >>>>>>> vulnerabilities; individual modules can always use a >>>>> different >>>>> >>> > >> version >>>>> >>> > >>>>>>> (that hopefully includes the fixes). >>>>> >>> > >>>>>>> Ideally of course we'd only be using 1 version, but that >>>>> may >>>>> >>> or may >>>>> >>> > >> not >>>>> >>> > >>>>>>> be feasible. >>>>> >>> > >>>>>>> >>>>> >>> > >>>>>>> On 15/11/2019 04:07, Hequn Cheng wrote: >>>>> >>> > >>>>>>>> Hi Chesnay, >>>>> >>> > >>>>>>>> >>>>> >>> > >>>>>>>> Great to hear that jackson-2.10.1 works well on master. >>>>> >>> Really a >>>>> >>> > >> good >>>>> >>> > >>>>>> job! >>>>> >>> > >>>>>>>> - Whether backport this change to 1.8/1.9 >>>>> >>> > >>>>>>>> I had taken a quick look at the security >>>>> vulnerabilities, >>>>> >>> some of >>>>> >>> > >> them >>>>> >>> > >>>>>>>> seem can lead to high-security problems, thus from my >>>>> point of >>>>> >>> > view, >>>>> >>> > >>>>>>>> I'm in favor of adding the fix into 1.9/1.8. However, I >>>>> would >>>>> >>> like >>>>> >>> > >> to >>>>> >>> > >>>>>>>> trust your judgment as you are more professional at this >>>>> >>> problem. >>>>> >>> > >>>>>>>> >>>>> >>> > >>>>>>>> - How to port this change to 1.8/1.9 >>>>> >>> > >>>>>>>> I think providing an opt-in upgrade is a good idea. >>>>> Another >>>>> >>> > question >>>>> >>> > >>>>>>>> here is whether do we plan to support multi jackson >>>>> versions >>>>> >>> that >>>>> >>> > >> have >>>>> >>> > >>>>>>>> eliminated the security vulnerabilities. If we only >>>>> plan to >>>>> >>> > support >>>>> >>> > >>>>>>>> 2.10.1, I would like to make it a non-opt-in upgrade. >>>>> As an >>>>> >>> > option, >>>>> >>> > >>>>>>>> users can downgrade the flink version if meet problems >>>>> using >>>>> >>> the >>>>> >>> > new >>>>> >>> > >>>>>>>> version. Of course, we will try our best to make the new >>>>> >>> release >>>>> >>> > out >>>>> >>> > >>>>>>>> of question. >>>>> >>> > >>>>>>>> Another concern of making it an opt-in upgrade is, it >>>>> will >>>>> >>> make >>>>> >>> > our >>>>> >>> > >>>>>>>> build unlikely convergence as more and more build >>>>> options >>>>> >>> will be >>>>> >>> > >>>>>>>> added when we upgrade a commonly used lib like this one. >>>>> >>> > >>>>>>>> >>>>> >>> > >>>>>>>> What do you think? >>>>> >>> > >>>>>>>> >>>>> >>> > >>>>>>>> Best, Hequn >>>>> >>> > >>>>>>>> >>>>> >>> > >>>>>>>> On Thu, Nov 14, 2019 at 6:00 PM Chesnay Schepler < >>>>> >>> > >> ches...@apache.org >>>>> >>> > >>>>>>>> <mailto:ches...@apache.org>> wrote: >>>>> >>> > >>>>>>>> >>>>> >>> > >>>>>>>> So here's the state of things: >>>>> >>> > >>>>>>>> >>>>> >>> > >>>>>>>> >>>>> >>> > >>>>>>>> The master of flink-shaded now uses jackson >>>>> 2.10.1, >>>>> >>> which >>>>> >>> > >>>>>>>> eliminates a whole category of security >>>>> >>> vulnerabilities. >>>>> >>> > >>>>>>>> The flink master works perfectly fine with that >>>>> >>> version; >>>>> >>> > 1.9 >>>>> >>> > >> will >>>>> >>> > >>>>>>>> likely do so too and 1.8 would require a minor >>>>> >>> adjustment. >>>>> >>> > >>>>>>>> >>>>> >>> > >>>>>>>> Hence, there may be value in first doing a >>>>> flink-shaded >>>>> >>> > >>>>>>>> release so >>>>> >>> > >>>>>>>> we can eliminate these vulnerabilities in 1.8.3 >>>>> and >>>>> >>> 1.9.2 . >>>>> >>> > >>>>>>>> >>>>> >>> > >>>>>>>> >>>>> >>> > >>>>>>>> As for other jackson dependencies (coming from >>>>> calcite, >>>>> >>> > kafka, >>>>> >>> > >>>>>>>> kinesis), I ran the unit and end-to-end tests of >>>>> master >>>>> >>> > >> yesterday >>>>> >>> > >>>>>>>> will /all /jackson dependencies set to 2.10.1, >>>>> and they >>>>> >>> > >> passed. I >>>>> >>> > >>>>>>>> will open a PR soon-ish for making this change on >>>>> >>> master. >>>>> >>> > >>>>>>>> >>>>> >>> > >>>>>>>> The question now is whether we want to backport >>>>> this >>>>> >>> > change to >>>>> >>> > >>>>>>>> 1.8/1.9 . >>>>> >>> > >>>>>>>> Some code paths /may /not be covered by our >>>>> tests, and >>>>> >>> > >> transitive >>>>> >>> > >>>>>>>> jackson users /might /run into issues. >>>>> >>> > >>>>>>>> Alternatively, we could set this up as an opt-in >>>>> >>> upgrade, >>>>> >>> > by >>>>> >>> > >>>>>>>> adding a separate profile that bumps the >>>>> versions. This >>>>> >>> > would >>>>> >>> > >>>>>>>> present users/providers who are concerned about >>>>> the >>>>> >>> > >>>>>>>> vulnerabilities an easy workaround, at the risk >>>>> of >>>>> >>> /some >>>>> >>> > >> /things >>>>> >>> > >>>>>>>> /maybe /not working. >>>>> >>> > >>>>>>>> >>>>> >>> > >>>>>>>> On 14/11/2019 03:16, Hequn Cheng wrote: >>>>> >>> > >>>>>>>>> Hi Chesnay, Jincheng >>>>> >>> > >>>>>>>>> >>>>> >>> > >>>>>>>>> Sure, I think it's good to have these fixes. >>>>> >>> > >>>>>>>>> Thanks a lot for providing the information >>>>> about the >>>>> >>> > security >>>>> >>> > >>>>>>>>> vulnerabilities! @Chesnay >>>>> >>> > >>>>>>>>> >>>>> >>> > >>>>>>>>> Best, Hequn >>>>> >>> > >>>>>>>>> >>>>> >>> > >>>>>>>>> On Thu, Nov 14, 2019 at 10:07 AM jincheng sun< >>>>> >>> > >>>>>> sunjincheng...@gmail.com> <mailto: >>>>> sunjincheng...@gmail.com> >>>>> >>> > >>>>>>>>> wrote: >>>>> >>> > >>>>>>>>> >>>>> >>> > >>>>>>>>>> +1 for try to eliminate the security >>>>> vulnerabilities. >>>>> >>> > Great >>>>> >>> > >>>>>> thanks for >>>>> >>> > >>>>>>>>>> doing this important work, Chesnay! >>>>> >>> > >>>>>>>>>> What do you think Hequn ? >>>>> >>> > >>>>>>>>>> >>>>> >>> > >>>>>>>>>> Best, >>>>> >>> > >>>>>>>>>> Jincheng >>>>> >>> > >>>>>>>>>> >>>>> >>> > >>>>>>>>>> Chesnay Schepler<ches...@apache.org> >>>>> >>> > >>>>>>>>>> <mailto:ches...@apache.org> >>>>> >>> > >>>>>> 于2019年11月13日周三 下午5:17写道： >>>>> >>> > >>>>>>>>>>> It would be great if you could give me a day >>>>> or 2 to >>>>> >>> > check >>>>> >>> > >> how >>>>> >>> > >>>>>> easy it >>>>> >>> > >>>>>>>>>>> would be to bump the various jackson >>>>> dependencies to >>>>> >>> > >>>>>>>>>>> eliminate a >>>>> >>> > >>>>>> few >>>>> >>> > >>>>>>>>>>> security vulnerabilities. >>>>> >>> > >>>>>>>>>>> >>>>> >>> > >>>>>>>>>>> On 09/11/2019 05:10, jincheng sun wrote: >>>>> >>> > >>>>>>>>>>>> Hi Flink devs, >>>>> >>> > >>>>>>>>>>>> >>>>> >>> > >>>>>>>>>>>> It has been more than 2 months since the >>>>> 1.8.2 >>>>> >>> > released. >>>>> >>> > >> So, >>>>> >>> > >>>>>> What do >>>>> >>> > >>>>>>>>>> you >>>>> >>> > >>>>>>>>>>>> think about releasing Flink 1.8.3 soon? >>>>> >>> > >>>>>>>>>>>> >>>>> >>> > >>>>>>>>>>>> We already have many important bug fixes in >>>>> the >>>>> >>> > >> release-1.8 >>>>> >>> > >>>>>> branch (29 >>>>> >>> > >>>>>>>>>>>> resolved issues). >>>>> >>> > >>>>>>>>>>>> >>>>> >>> > >>>>>>>>>>>> Most notable fixes are: >>>>> >>> > >>>>>>>>>>>> >>>>> >>> > >>>>>>>>>>>> - FLINK-14010 Dispatcher & JobManagers don't >>>>> give >>>>> >>> up >>>>> >>> > >>>>>>>>>>>> leadership >>>>> >>> > >>>>>> when AM >>>>> >>> > >>>>>>>>>>> is >>>>> >>> > >>>>>>>>>>>> shut down >>>>> >>> > >>>>>>>>>>>> - FLINK-14315 NPE with >>>>> >>> JobMaster.disconnectTaskManager >>>>> >>> > >>>>>>>>>>>> - FLINK-12848 Method equals() in RowTypeInfo >>>>> should >>>>> >>> > >> consider >>>>> >>> > >>>>>>>>>> fieldsNames >>>>> >>> > >>>>>>>>>>>> - FLINK-12342 Yarn Resource Manager Acquires >>>>> Too >>>>> >>> Many >>>>> >>> > >>>>>>>>>>>> Containers >>>>> >>> > >>>>>>>>>>>> - FLINK-14589 Redundant slot requests with >>>>> the same >>>>> >>> > >>>>>> AllocationID leads >>>>> >>> > >>>>>>>>>> to >>>>> >>> > >>>>>>>>>>>> inconsistent slot table >>>>> >>> > >>>>>>>>>>>> >>>>> >>> > >>>>>>>>>>>> Furthermore, the following critical issues >>>>> is in >>>>> >>> > progress, >>>>> >>> > >>>>>> maybe we can >>>>> >>> > >>>>>>>>>>>> wait for it if it is not too much effort. >>>>> >>> > >>>>>>>>>>>> >>>>> >>> > >>>>>>>>>>>> - FLINK-13184 Starting a TaskExecutor blocks >>>>> the >>>>> >>> > >>>>>> YarnResourceManager's >>>>> >>> > >>>>>>>>>>> main >>>>> >>> > >>>>>>>>>>>> thread >>>>> >>> > >>>>>>>>>>>> >>>>> >>> > >>>>>>>>>>>> Please let me know what you think? >>>>> >>> > >>>>>>>>>>>> >>>>> >>> > >>>>>>>>>>>> Best, >>>>> >>> > >>>>>>>>>>>> Jincheng >>>>> >>> > >>>>>>>>>>>> >>>>> >>> > >> >>>>> >>> > >>>>> >>> > >>>>> >>> >>>>> >> >>>>> >>>>
