Hi Hequn, Thank you for your great job! Looking forward the first RC of 1.8.3 ! BTW: The version of 1.8.4 already created here: https://issues.apache.org/jira/projects/FLINK/versions/12346552
Best, Jincheng Hequn Cheng <chenghe...@gmail.com> 于2019年11月26日周二 下午8:18写道： > Hi all, > > I would like to share with you that all blockers are resolved now. If > there are no more critical issues, I will create the first RC tomorrow and > vote on it directly. > Hope everything goes well! > > Thank you all for the help of fixing, reviewing, driving and discussions! > > Best, Hequn > > On Tue, Nov 26, 2019 at 9:27 AM Hequn Cheng <chenghe...@gmail.com> wrote: > >> Hi, >> >> @Ufuk Celebi <u...@apache.org> Hi, we are very close now. There is one >> issue(FLINK-13995 <https://issues.apache.org/jira/browse/FLINK-13995>) >> left that I want to double-check with you guys. Once this is done, we can >> create the first RC. I already have some minor comments in the PR >> <https://github.com/apache/flink/pull/10195>. >> >> @Zhu Zhu <reed...@gmail.com> Glad to hear that it is not a blocker. >> Thank you. >> >> Best, Hequn >> >> On Mon, Nov 25, 2019 at 5:43 PM Ufuk Celebi <u...@apache.org> wrote: >> >>> @Hequn: flink-shaded:9.0 is available in Maven central now. I think you >>> can go ahead and create the first RC. :-) >>> >>> On Mon, Nov 25, 2019 at 7:47 AM Zhu Zhu <reed...@gmail.com> wrote: >>> >>>> Hi Hequn, >>>> >>>> Looks we are not able to merge fix of FLINK-14735 to 1.8 very soon. >>>> Given that this fix is for batch job only and batch is not very good in >>>> 1.8, I think it is a not blocker of release 1.8.3. >>>> So just don't be blocked by it and feel free to cut the RC when other >>>> blocking issues are resolved. >>>> >>>> Thanks, >>>> Zhu Zhu >>>> >>>> Hequn Cheng <chenghe...@gmail.com> 于2019年11月23日周六 下午9:08写道： >>>> >>>> > Hi Zhu Zhu, >>>> > >>>> > Thanks a lot for letting us know! >>>> > We can't cut the first RC right now due to the wait of the flink-shade >>>> > release, so go ahead. >>>> > >>>> > Theoretically, we will cut the first RC of 1.8.3 and vote for it once >>>> the >>>> > release of flink-shade is done, >>>> > but I will try my best to have it in 1.8.3. Hope we can get it on >>>> board on >>>> > time. :) >>>> > >>>> > Best, Hequn >>>> > >>>> > On Sat, Nov 23, 2019 at 10:40 AM Zhu Zhu <reed...@gmail.com> wrote: >>>> > >>>> >> Hi Jincheng & Hequn >>>> >> >>>> >> Thanks for driving the releasing of 1.8.3. >>>> >> >>>> >> I am now working on FLINK-14735. The fix avoids duplicated input >>>> >> checking when scheduling ALL-to-ALL >>>> >> connected downstream consumers with ALL input constraints. The >>>> duplicated >>>> >> checking can cause severe >>>> >> performance issues for large scale jobs. So I hope the fix could be >>>> >> released with 1.8.3. >>>> >> >>>> >> The fix is already merged into master, and is now in the process of >>>> >> backporting to 1.8. >>>> >> >>>> >> Thanks, >>>> >> Zhu Zhu >>>> >> >>>> >> Ufuk Celebi <u...@apache.org> 于2019年11月15日周五 下午11:54写道： >>>> >> >>>> >>> Thanks Chesnay. >>>> >>> >>>> >>> I'm also +1 to release 1.8.3 asap without the changes for the >>>> Jackson >>>> >>> version bump and leave those for a future release. Realistically, >>>> the >>>> >>> flink-shaded release will take until mid next week or end of next >>>> week. >>>> >>> But >>>> >>> please correct me if you think that it should not take that long or >>>> it's >>>> >>> OK >>>> >>> to block the 1.8.3 release on the flink-shaded release. >>>> >>> >>>> >>> – Ufuk >>>> >>> >>>> >>> >>>> >>> On Fri, Nov 15, 2019 at 2:27 PM Chesnay Schepler < >>>> ches...@apache.org> >>>> >>> wrote: >>>> >>> >>>> >>> > I've kicked off a discussion about the next flink-shaded release, >>>> and >>>> >>> > have opened PRs for adding the opt-in profile to 1.8/1.9. >>>> >>> > >>>> >>> > On 15/11/2019 13:54, Hequn Cheng wrote: >>>> >>> > > That's great, thank you very much! Ideally, we can kick off the >>>> >>> release >>>> >>> > > vote for the first RC of 1.8.3 within next week. :) >>>> >>> > > >>>> >>> > > On Fri, Nov 15, 2019 at 8:47 PM Chesnay Schepler < >>>> ches...@apache.org >>>> >>> > >>>> >>> > wrote: >>>> >>> > > >>>> >>> > >> I'm not aware of any more planned changes to flink-shaded; so >>>> we >>>> >>> could >>>> >>> > >> start the release right away. >>>> >>> > >> >>>> >>> > >> On 15/11/2019 13:44, Hequn Cheng wrote: >>>> >>> > >>> Hi, >>>> >>> > >>> >>>> >>> > >>> @Chesnay Thanks a lot for the explanation. +1 to the opt-in >>>> >>> approach >>>> >>> > for >>>> >>> > >>> 1.8/1.9. >>>> >>> > >>> @Ufuk Thank you for the nice summary. >>>> >>> > >>> >>>> >>> > >>> Looks good so far except that we need to postpone 1.8.3 a bit >>>> to >>>> >>> first >>>> >>> > >> do a >>>> >>> > >>> flink-shaded release. >>>> >>> > >>> BTW, @chesnay when would we plan to release the flink-shaded >>>> with >>>> >>> > >> upgraded >>>> >>> > >>> Jackson? >>>> >>> > >>> >>>> >>> > >>> Best, Hequn >>>> >>> > >>> >>>> >>> > >>> On Fri, Nov 15, 2019 at 7:43 PM Chesnay Schepler < >>>> >>> ches...@apache.org> >>>> >>> > >> wrote: >>>> >>> > >>>> One small modification: the flink-shaded upgrade does not >>>> have to >>>> >>> be >>>> >>> > >>>> part of the profile; since it is only intended for internal >>>> use >>>> >>> anyway >>>> >>> > >>>> (and thus has limited exposure) we can be pretty sure this >>>> doesn't >>>> >>> > break >>>> >>> > >>>> anything. >>>> >>> > >>>> >>>> >>> > >>>> On 15/11/2019 12:23, Chesnay Schepler wrote: >>>> >>> > >>>>> Ufuk's summary is correct. >>>> >>> > >>>>> >>>> >>> > >>>>> There's a slight caveat in that we'd also have to bump the >>>> >>> > >>>>> shade-plugin to 3.1.1 since it otherwise fails on jackson, >>>> >>> > >>>>> but I have no concerns about this change. >>>> >>> > >>>>> >>>> >>> > >>>>> On 15/11/2019 12:19, Ufuk Celebi wrote: >>>> >>> > >>>>>> The opt-in approach seems reasonable to me. +1 to include >>>> the >>>> >>> > >>>>>> profiles in >>>> >>> > >>>>>> 1.8 and 1.9 without changing the default versions >>>> (including the >>>> >>> > >> default >>>> >>> > >>>>>> version of flink-shaded). >>>> >>> > >>>>>> >>>> >>> > >>>>>> As far as I can tell, the next steps would be: >>>> >>> > >>>>>> >>>> >>> > >>>>>> 1) Release flink-shaded with upgraded Jackson >>>> >>> > >>>>>> 2a) Bump the flink-shaded version by default in master >>>> >>> > >>>>>> 2b) Create opt-in profiles for 1.8 and 1.9 (the opt-in >>>> profiles >>>> >>> > >>>>>> should also >>>> >>> > >>>>>> cover the upgrade to the most recent flink-shaded version) >>>> >>> > >>>>>> >>>> >>> > >>>>>> @Chesnay: is this a correct summary? >>>> >>> > >>>>>> >>>> >>> > >>>>>> Note this would block the 1.8.3 release on step 1. As an >>>> >>> upside, we >>>> >>> > >>>>>> might >>>> >>> > >>>>>> get some additional feedback until the 1.10 release with >>>> these >>>> >>> > >>>>>> profiles in >>>> >>> > >>>>>> case users make use of them with 1.8/1.9. >>>> >>> > >>>>>> >>>> >>> > >>>>>> – Ufuk >>>> >>> > >>>>>> >>>> >>> > >>>>>> On Fri, Nov 15, 2019 at 12:08 PM Chesnay Schepler < >>>> >>> > ches...@apache.org >>>> >>> > >>>>>> wrote: >>>> >>> > >>>>>>> The opt-in approach would only be used for 1.8.3 / 1.9.2; >>>> on >>>> >>> master >>>> >>> > >>>>>>> (and >>>> >>> > >>>>>>> thus starting from 1.10.0) it's not opt-in. >>>> >>> > >>>>>>> >>>> >>> > >>>>>>> I have only proposed it as an opt-in because a) we >>>> usually do >>>> >>> not >>>> >>> > >> bump >>>> >>> > >>>>>>> dependencies in bugfix releases and b) it's a short-term >>>> change >>>> >>> > that >>>> >>> > >> we >>>> >>> > >>>>>>> aren't allowing to mature properly. >>>> >>> > >>>>>>> In contrast, the 1.10 release is significantly further >>>> away, >>>> >>> hence >>>> >>> > no >>>> >>> > >>>>>>> opt-in. >>>> >>> > >>>>>>> >>>> >>> > >>>>>>> Hence, I'm not concerned about such kind of ugprades >>>> being more >>>> >>> > >> common >>>> >>> > >>>>>>> in the future. >>>> >>> > >>>>>>> >>>> >>> > >>>>>>> We can certainly support every jackson version that fixes >>>> these >>>> >>> > >>>>>>> vulnerabilities; individual modules can always use a >>>> different >>>> >>> > >> version >>>> >>> > >>>>>>> (that hopefully includes the fixes). >>>> >>> > >>>>>>> Ideally of course we'd only be using 1 version, but that >>>> may >>>> >>> or may >>>> >>> > >> not >>>> >>> > >>>>>>> be feasible. >>>> >>> > >>>>>>> >>>> >>> > >>>>>>> On 15/11/2019 04:07, Hequn Cheng wrote: >>>> >>> > >>>>>>>> Hi Chesnay, >>>> >>> > >>>>>>>> >>>> >>> > >>>>>>>> Great to hear that jackson-2.10.1 works well on master. >>>> >>> Really a >>>> >>> > >> good >>>> >>> > >>>>>> job! >>>> >>> > >>>>>>>> - Whether backport this change to 1.8/1.9 >>>> >>> > >>>>>>>> I had taken a quick look at the security vulnerabilities, >>>> >>> some of >>>> >>> > >> them >>>> >>> > >>>>>>>> seem can lead to high-security problems, thus from my >>>> point of >>>> >>> > view, >>>> >>> > >>>>>>>> I'm in favor of adding the fix into 1.9/1.8. However, I >>>> would >>>> >>> like >>>> >>> > >> to >>>> >>> > >>>>>>>> trust your judgment as you are more professional at this >>>> >>> problem. >>>> >>> > >>>>>>>> >>>> >>> > >>>>>>>> - How to port this change to 1.8/1.9 >>>> >>> > >>>>>>>> I think providing an opt-in upgrade is a good idea. >>>> Another >>>> >>> > question >>>> >>> > >>>>>>>> here is whether do we plan to support multi jackson >>>> versions >>>> >>> that >>>> >>> > >> have >>>> >>> > >>>>>>>> eliminated the security vulnerabilities. If we only plan >>>> to >>>> >>> > support >>>> >>> > >>>>>>>> 2.10.1, I would like to make it a non-opt-in upgrade. As >>>> an >>>> >>> > option, >>>> >>> > >>>>>>>> users can downgrade the flink version if meet problems >>>> using >>>> >>> the >>>> >>> > new >>>> >>> > >>>>>>>> version. Of course, we will try our best to make the new >>>> >>> release >>>> >>> > out >>>> >>> > >>>>>>>> of question. >>>> >>> > >>>>>>>> Another concern of making it an opt-in upgrade is, it >>>> will >>>> >>> make >>>> >>> > our >>>> >>> > >>>>>>>> build unlikely convergence as more and more build options >>>> >>> will be >>>> >>> > >>>>>>>> added when we upgrade a commonly used lib like this one. >>>> >>> > >>>>>>>> >>>> >>> > >>>>>>>> What do you think? >>>> >>> > >>>>>>>> >>>> >>> > >>>>>>>> Best, Hequn >>>> >>> > >>>>>>>> >>>> >>> > >>>>>>>> On Thu, Nov 14, 2019 at 6:00 PM Chesnay Schepler < >>>> >>> > >> ches...@apache.org >>>> >>> > >>>>>>>> <mailto:ches...@apache.org>> wrote: >>>> >>> > >>>>>>>> >>>> >>> > >>>>>>>> So here's the state of things: >>>> >>> > >>>>>>>> >>>> >>> > >>>>>>>> >>>> >>> > >>>>>>>> The master of flink-shaded now uses jackson >>>> 2.10.1, >>>> >>> which >>>> >>> > >>>>>>>> eliminates a whole category of security >>>> >>> vulnerabilities. >>>> >>> > >>>>>>>> The flink master works perfectly fine with that >>>> >>> version; >>>> >>> > 1.9 >>>> >>> > >> will >>>> >>> > >>>>>>>> likely do so too and 1.8 would require a minor >>>> >>> adjustment. >>>> >>> > >>>>>>>> >>>> >>> > >>>>>>>> Hence, there may be value in first doing a >>>> flink-shaded >>>> >>> > >>>>>>>> release so >>>> >>> > >>>>>>>> we can eliminate these vulnerabilities in 1.8.3 >>>> and >>>> >>> 1.9.2 . >>>> >>> > >>>>>>>> >>>> >>> > >>>>>>>> >>>> >>> > >>>>>>>> As for other jackson dependencies (coming from >>>> calcite, >>>> >>> > kafka, >>>> >>> > >>>>>>>> kinesis), I ran the unit and end-to-end tests of >>>> master >>>> >>> > >> yesterday >>>> >>> > >>>>>>>> will /all /jackson dependencies set to 2.10.1, >>>> and they >>>> >>> > >> passed. I >>>> >>> > >>>>>>>> will open a PR soon-ish for making this change on >>>> >>> master. >>>> >>> > >>>>>>>> >>>> >>> > >>>>>>>> The question now is whether we want to backport >>>> this >>>> >>> > change to >>>> >>> > >>>>>>>> 1.8/1.9 . >>>> >>> > >>>>>>>> Some code paths /may /not be covered by our >>>> tests, and >>>> >>> > >> transitive >>>> >>> > >>>>>>>> jackson users /might /run into issues. >>>> >>> > >>>>>>>> Alternatively, we could set this up as an opt-in >>>> >>> upgrade, >>>> >>> > by >>>> >>> > >>>>>>>> adding a separate profile that bumps the >>>> versions. This >>>> >>> > would >>>> >>> > >>>>>>>> present users/providers who are concerned about >>>> the >>>> >>> > >>>>>>>> vulnerabilities an easy workaround, at the risk of >>>> >>> /some >>>> >>> > >> /things >>>> >>> > >>>>>>>> /maybe /not working. >>>> >>> > >>>>>>>> >>>> >>> > >>>>>>>> On 14/11/2019 03:16, Hequn Cheng wrote: >>>> >>> > >>>>>>>>> Hi Chesnay, Jincheng >>>> >>> > >>>>>>>>> >>>> >>> > >>>>>>>>> Sure, I think it's good to have these fixes. >>>> >>> > >>>>>>>>> Thanks a lot for providing the information about >>>> the >>>> >>> > security >>>> >>> > >>>>>>>>> vulnerabilities! @Chesnay >>>> >>> > >>>>>>>>> >>>> >>> > >>>>>>>>> Best, Hequn >>>> >>> > >>>>>>>>> >>>> >>> > >>>>>>>>> On Thu, Nov 14, 2019 at 10:07 AM jincheng sun< >>>> >>> > >>>>>> sunjincheng...@gmail.com> <mailto:sunjincheng...@gmail.com >>>> > >>>> >>> > >>>>>>>>> wrote: >>>> >>> > >>>>>>>>> >>>> >>> > >>>>>>>>>> +1 for try to eliminate the security >>>> vulnerabilities. >>>> >>> > Great >>>> >>> > >>>>>> thanks for >>>> >>> > >>>>>>>>>> doing this important work, Chesnay! >>>> >>> > >>>>>>>>>> What do you think Hequn ? >>>> >>> > >>>>>>>>>> >>>> >>> > >>>>>>>>>> Best, >>>> >>> > >>>>>>>>>> Jincheng >>>> >>> > >>>>>>>>>> >>>> >>> > >>>>>>>>>> Chesnay Schepler<ches...@apache.org> >>>> >>> > >>>>>>>>>> <mailto:ches...@apache.org> >>>> >>> > >>>>>> 于2019年11月13日周三 下午5:17写道： >>>> >>> > >>>>>>>>>>> It would be great if you could give me a day >>>> or 2 to >>>> >>> > check >>>> >>> > >> how >>>> >>> > >>>>>> easy it >>>> >>> > >>>>>>>>>>> would be to bump the various jackson >>>> dependencies to >>>> >>> > >>>>>>>>>>> eliminate a >>>> >>> > >>>>>> few >>>> >>> > >>>>>>>>>>> security vulnerabilities. >>>> >>> > >>>>>>>>>>> >>>> >>> > >>>>>>>>>>> On 09/11/2019 05:10, jincheng sun wrote: >>>> >>> > >>>>>>>>>>>> Hi Flink devs, >>>> >>> > >>>>>>>>>>>> >>>> >>> > >>>>>>>>>>>> It has been more than 2 months since the 1.8.2 >>>> >>> > released. >>>> >>> > >> So, >>>> >>> > >>>>>> What do >>>> >>> > >>>>>>>>>> you >>>> >>> > >>>>>>>>>>>> think about releasing Flink 1.8.3 soon? >>>> >>> > >>>>>>>>>>>> >>>> >>> > >>>>>>>>>>>> We already have many important bug fixes in >>>> the >>>> >>> > >> release-1.8 >>>> >>> > >>>>>> branch (29 >>>> >>> > >>>>>>>>>>>> resolved issues). >>>> >>> > >>>>>>>>>>>> >>>> >>> > >>>>>>>>>>>> Most notable fixes are: >>>> >>> > >>>>>>>>>>>> >>>> >>> > >>>>>>>>>>>> - FLINK-14010 Dispatcher & JobManagers don't >>>> give >>>> >>> up >>>> >>> > >>>>>>>>>>>> leadership >>>> >>> > >>>>>> when AM >>>> >>> > >>>>>>>>>>> is >>>> >>> > >>>>>>>>>>>> shut down >>>> >>> > >>>>>>>>>>>> - FLINK-14315 NPE with >>>> >>> JobMaster.disconnectTaskManager >>>> >>> > >>>>>>>>>>>> - FLINK-12848 Method equals() in RowTypeInfo >>>> should >>>> >>> > >> consider >>>> >>> > >>>>>>>>>> fieldsNames >>>> >>> > >>>>>>>>>>>> - FLINK-12342 Yarn Resource Manager Acquires >>>> Too >>>> >>> Many >>>> >>> > >>>>>>>>>>>> Containers >>>> >>> > >>>>>>>>>>>> - FLINK-14589 Redundant slot requests with >>>> the same >>>> >>> > >>>>>> AllocationID leads >>>> >>> > >>>>>>>>>> to >>>> >>> > >>>>>>>>>>>> inconsistent slot table >>>> >>> > >>>>>>>>>>>> >>>> >>> > >>>>>>>>>>>> Furthermore, the following critical issues is >>>> in >>>> >>> > progress, >>>> >>> > >>>>>> maybe we can >>>> >>> > >>>>>>>>>>>> wait for it if it is not too much effort. >>>> >>> > >>>>>>>>>>>> >>>> >>> > >>>>>>>>>>>> - FLINK-13184 Starting a TaskExecutor blocks >>>> the >>>> >>> > >>>>>> YarnResourceManager's >>>> >>> > >>>>>>>>>>> main >>>> >>> > >>>>>>>>>>>> thread >>>> >>> > >>>>>>>>>>>> >>>> >>> > >>>>>>>>>>>> Please let me know what you think? >>>> >>> > >>>>>>>>>>>> >>>> >>> > >>>>>>>>>>>> Best, >>>> >>> > >>>>>>>>>>>> Jincheng >>>> >>> > >>>>>>>>>>>> >>>> >>> > >> >>>> >>> > >>>> >>> > >>>> >>> >>>> >> >>>> >>>
