Hi Jincheng & Hequn Thanks for driving the releasing of 1.8.3.
I am now working on FLINK-14735. The fix avoids duplicated input checking when scheduling ALL-to-ALL connected downstream consumers with ALL input constraints. The duplicated checking can cause severe performance issues for large scale jobs. So I hope the fix could be released with 1.8.3. The fix is already merged into master, and is now in the process of backporting to 1.8. Thanks, Zhu Zhu Ufuk Celebi <u...@apache.org> 于2019年11月15日周五 下午11:54写道： > Thanks Chesnay. > > I'm also +1 to release 1.8.3 asap without the changes for the Jackson > version bump and leave those for a future release. Realistically, the > flink-shaded release will take until mid next week or end of next week. But > please correct me if you think that it should not take that long or it's OK > to block the 1.8.3 release on the flink-shaded release. > > – Ufuk > > > On Fri, Nov 15, 2019 at 2:27 PM Chesnay Schepler <ches...@apache.org> > wrote: > > > I've kicked off a discussion about the next flink-shaded release, and > > have opened PRs for adding the opt-in profile to 1.8/1.9. > > > > On 15/11/2019 13:54, Hequn Cheng wrote: > > > That's great, thank you very much! Ideally, we can kick off the release > > > vote for the first RC of 1.8.3 within next week. :) > > > > > > On Fri, Nov 15, 2019 at 8:47 PM Chesnay Schepler <ches...@apache.org> > > wrote: > > > > > >> I'm not aware of any more planned changes to flink-shaded; so we could > > >> start the release right away. > > >> > > >> On 15/11/2019 13:44, Hequn Cheng wrote: > > >>> Hi, > > >>> > > >>> @Chesnay Thanks a lot for the explanation. +1 to the opt-in approach > > for > > >>> 1.8/1.9. > > >>> @Ufuk Thank you for the nice summary. > > >>> > > >>> Looks good so far except that we need to postpone 1.8.3 a bit to > first > > >> do a > > >>> flink-shaded release. > > >>> BTW, @chesnay when would we plan to release the flink-shaded with > > >> upgraded > > >>> Jackson? > > >>> > > >>> Best, Hequn > > >>> > > >>> On Fri, Nov 15, 2019 at 7:43 PM Chesnay Schepler <ches...@apache.org > > > > >> wrote: > > >>>> One small modification: the flink-shaded upgrade does not have to be > > >>>> part of the profile; since it is only intended for internal use > anyway > > >>>> (and thus has limited exposure) we can be pretty sure this doesn't > > break > > >>>> anything. > > >>>> > > >>>> On 15/11/2019 12:23, Chesnay Schepler wrote: > > >>>>> Ufuk's summary is correct. > > >>>>> > > >>>>> There's a slight caveat in that we'd also have to bump the > > >>>>> shade-plugin to 3.1.1 since it otherwise fails on jackson, > > >>>>> but I have no concerns about this change. > > >>>>> > > >>>>> On 15/11/2019 12:19, Ufuk Celebi wrote: > > >>>>>> The opt-in approach seems reasonable to me. +1 to include the > > >>>>>> profiles in > > >>>>>> 1.8 and 1.9 without changing the default versions (including the > > >> default > > >>>>>> version of flink-shaded). > > >>>>>> > > >>>>>> As far as I can tell, the next steps would be: > > >>>>>> > > >>>>>> 1) Release flink-shaded with upgraded Jackson > > >>>>>> 2a) Bump the flink-shaded version by default in master > > >>>>>> 2b) Create opt-in profiles for 1.8 and 1.9 (the opt-in profiles > > >>>>>> should also > > >>>>>> cover the upgrade to the most recent flink-shaded version) > > >>>>>> > > >>>>>> @Chesnay: is this a correct summary? > > >>>>>> > > >>>>>> Note this would block the 1.8.3 release on step 1. As an upside, > we > > >>>>>> might > > >>>>>> get some additional feedback until the 1.10 release with these > > >>>>>> profiles in > > >>>>>> case users make use of them with 1.8/1.9. > > >>>>>> > > >>>>>> – Ufuk > > >>>>>> > > >>>>>> On Fri, Nov 15, 2019 at 12:08 PM Chesnay Schepler < > > ches...@apache.org > > >>>>>> wrote: > > >>>>>>> The opt-in approach would only be used for 1.8.3 / 1.9.2; on > master > > >>>>>>> (and > > >>>>>>> thus starting from 1.10.0) it's not opt-in. > > >>>>>>> > > >>>>>>> I have only proposed it as an opt-in because a) we usually do not > > >> bump > > >>>>>>> dependencies in bugfix releases and b) it's a short-term change > > that > > >> we > > >>>>>>> aren't allowing to mature properly. > > >>>>>>> In contrast, the 1.10 release is significantly further away, > hence > > no > > >>>>>>> opt-in. > > >>>>>>> > > >>>>>>> Hence, I'm not concerned about such kind of ugprades being more > > >> common > > >>>>>>> in the future. > > >>>>>>> > > >>>>>>> We can certainly support every jackson version that fixes these > > >>>>>>> vulnerabilities; individual modules can always use a different > > >> version > > >>>>>>> (that hopefully includes the fixes). > > >>>>>>> Ideally of course we'd only be using 1 version, but that may or > may > > >> not > > >>>>>>> be feasible. > > >>>>>>> > > >>>>>>> On 15/11/2019 04:07, Hequn Cheng wrote: > > >>>>>>>> Hi Chesnay, > > >>>>>>>> > > >>>>>>>> Great to hear that jackson-2.10.1 works well on master. Really a > > >> good > > >>>>>> job! > > >>>>>>>> - Whether backport this change to 1.8/1.9 > > >>>>>>>> I had taken a quick look at the security vulnerabilities, some > of > > >> them > > >>>>>>>> seem can lead to high-security problems, thus from my point of > > view, > > >>>>>>>> I'm in favor of adding the fix into 1.9/1.8. However, I would > like > > >> to > > >>>>>>>> trust your judgment as you are more professional at this > problem. > > >>>>>>>> > > >>>>>>>> - How to port this change to 1.8/1.9 > > >>>>>>>> I think providing an opt-in upgrade is a good idea. Another > > question > > >>>>>>>> here is whether do we plan to support multi jackson versions > that > > >> have > > >>>>>>>> eliminated the security vulnerabilities. If we only plan to > > support > > >>>>>>>> 2.10.1, I would like to make it a non-opt-in upgrade. As an > > option, > > >>>>>>>> users can downgrade the flink version if meet problems using the > > new > > >>>>>>>> version. Of course, we will try our best to make the new release > > out > > >>>>>>>> of question. > > >>>>>>>> Another concern of making it an opt-in upgrade is, it will make > > our > > >>>>>>>> build unlikely convergence as more and more build options will > be > > >>>>>>>> added when we upgrade a commonly used lib like this one. > > >>>>>>>> > > >>>>>>>> What do you think? > > >>>>>>>> > > >>>>>>>> Best, Hequn > > >>>>>>>> > > >>>>>>>> On Thu, Nov 14, 2019 at 6:00 PM Chesnay Schepler < > > >> ches...@apache.org > > >>>>>>>> <mailto:ches...@apache.org>> wrote: > > >>>>>>>> > > >>>>>>>> So here's the state of things: > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> The master of flink-shaded now uses jackson 2.10.1, which > > >>>>>>>> eliminates a whole category of security vulnerabilities. > > >>>>>>>> The flink master works perfectly fine with that version; > > 1.9 > > >> will > > >>>>>>>> likely do so too and 1.8 would require a minor > adjustment. > > >>>>>>>> > > >>>>>>>> Hence, there may be value in first doing a flink-shaded > > >>>>>>>> release so > > >>>>>>>> we can eliminate these vulnerabilities in 1.8.3 and > 1.9.2 . > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> As for other jackson dependencies (coming from calcite, > > kafka, > > >>>>>>>> kinesis), I ran the unit and end-to-end tests of master > > >> yesterday > > >>>>>>>> will /all /jackson dependencies set to 2.10.1, and they > > >> passed. I > > >>>>>>>> will open a PR soon-ish for making this change on master. > > >>>>>>>> > > >>>>>>>> The question now is whether we want to backport this > > change to > > >>>>>>>> 1.8/1.9 . > > >>>>>>>> Some code paths /may /not be covered by our tests, and > > >> transitive > > >>>>>>>> jackson users /might /run into issues. > > >>>>>>>> Alternatively, we could set this up as an opt-in upgrade, > > by > > >>>>>>>> adding a separate profile that bumps the versions. This > > would > > >>>>>>>> present users/providers who are concerned about the > > >>>>>>>> vulnerabilities an easy workaround, at the risk of /some > > >> /things > > >>>>>>>> /maybe /not working. > > >>>>>>>> > > >>>>>>>> On 14/11/2019 03:16, Hequn Cheng wrote: > > >>>>>>>>> Hi Chesnay, Jincheng > > >>>>>>>>> > > >>>>>>>>> Sure, I think it's good to have these fixes. > > >>>>>>>>> Thanks a lot for providing the information about the > > security > > >>>>>>>>> vulnerabilities! @Chesnay > > >>>>>>>>> > > >>>>>>>>> Best, Hequn > > >>>>>>>>> > > >>>>>>>>> On Thu, Nov 14, 2019 at 10:07 AM jincheng sun< > > >>>>>> sunjincheng...@gmail.com> <mailto:sunjincheng...@gmail.com> > > >>>>>>>>> wrote: > > >>>>>>>>> > > >>>>>>>>>> +1 for try to eliminate the security vulnerabilities. > > Great > > >>>>>> thanks for > > >>>>>>>>>> doing this important work, Chesnay! > > >>>>>>>>>> What do you think Hequn ? > > >>>>>>>>>> > > >>>>>>>>>> Best, > > >>>>>>>>>> Jincheng > > >>>>>>>>>> > > >>>>>>>>>> Chesnay Schepler<ches...@apache.org> > > >>>>>>>>>> <mailto:ches...@apache.org> > > >>>>>> 于2019年11月13日周三 下午5:17写道： > > >>>>>>>>>>> It would be great if you could give me a day or 2 to > > check > > >> how > > >>>>>> easy it > > >>>>>>>>>>> would be to bump the various jackson dependencies to > > >>>>>>>>>>> eliminate a > > >>>>>> few > > >>>>>>>>>>> security vulnerabilities. > > >>>>>>>>>>> > > >>>>>>>>>>> On 09/11/2019 05:10, jincheng sun wrote: > > >>>>>>>>>>>> Hi Flink devs, > > >>>>>>>>>>>> > > >>>>>>>>>>>> It has been more than 2 months since the 1.8.2 > > released. > > >> So, > > >>>>>> What do > > >>>>>>>>>> you > > >>>>>>>>>>>> think about releasing Flink 1.8.3 soon? > > >>>>>>>>>>>> > > >>>>>>>>>>>> We already have many important bug fixes in the > > >> release-1.8 > > >>>>>> branch (29 > > >>>>>>>>>>>> resolved issues). > > >>>>>>>>>>>> > > >>>>>>>>>>>> Most notable fixes are: > > >>>>>>>>>>>> > > >>>>>>>>>>>> - FLINK-14010 Dispatcher & JobManagers don't give up > > >>>>>>>>>>>> leadership > > >>>>>> when AM > > >>>>>>>>>>> is > > >>>>>>>>>>>> shut down > > >>>>>>>>>>>> - FLINK-14315 NPE with > JobMaster.disconnectTaskManager > > >>>>>>>>>>>> - FLINK-12848 Method equals() in RowTypeInfo should > > >> consider > > >>>>>>>>>> fieldsNames > > >>>>>>>>>>>> - FLINK-12342 Yarn Resource Manager Acquires Too Many > > >>>>>>>>>>>> Containers > > >>>>>>>>>>>> - FLINK-14589 Redundant slot requests with the same > > >>>>>> AllocationID leads > > >>>>>>>>>> to > > >>>>>>>>>>>> inconsistent slot table > > >>>>>>>>>>>> > > >>>>>>>>>>>> Furthermore, the following critical issues is in > > progress, > > >>>>>> maybe we can > > >>>>>>>>>>>> wait for it if it is not too much effort. > > >>>>>>>>>>>> > > >>>>>>>>>>>> - FLINK-13184 Starting a TaskExecutor blocks the > > >>>>>> YarnResourceManager's > > >>>>>>>>>>> main > > >>>>>>>>>>>> thread > > >>>>>>>>>>>> > > >>>>>>>>>>>> Please let me know what you think? > > >>>>>>>>>>>> > > >>>>>>>>>>>> Best, > > >>>>>>>>>>>> Jincheng > > >>>>>>>>>>>> > > >> > > > > >
