That's great, thank you very much! Ideally, we can kick off the release vote for the first RC of 1.8.3 within next week. :)
On Fri, Nov 15, 2019 at 8:47 PM Chesnay Schepler <ches...@apache.org> wrote: > I'm not aware of any more planned changes to flink-shaded; so we could > start the release right away. > > On 15/11/2019 13:44, Hequn Cheng wrote: > > Hi, > > > > @Chesnay Thanks a lot for the explanation. +1 to the opt-in approach for > > 1.8/1.9. > > @Ufuk Thank you for the nice summary. > > > > Looks good so far except that we need to postpone 1.8.3 a bit to first > do a > > flink-shaded release. > > BTW, @chesnay when would we plan to release the flink-shaded with > upgraded > > Jackson? > > > > Best, Hequn > > > > On Fri, Nov 15, 2019 at 7:43 PM Chesnay Schepler <ches...@apache.org> > wrote: > > > >> One small modification: the flink-shaded upgrade does not have to be > >> part of the profile; since it is only intended for internal use anyway > >> (and thus has limited exposure) we can be pretty sure this doesn't break > >> anything. > >> > >> On 15/11/2019 12:23, Chesnay Schepler wrote: > >>> Ufuk's summary is correct. > >>> > >>> There's a slight caveat in that we'd also have to bump the > >>> shade-plugin to 3.1.1 since it otherwise fails on jackson, > >>> but I have no concerns about this change. > >>> > >>> On 15/11/2019 12:19, Ufuk Celebi wrote: > >>>> The opt-in approach seems reasonable to me. +1 to include the > >>>> profiles in > >>>> 1.8 and 1.9 without changing the default versions (including the > default > >>>> version of flink-shaded). > >>>> > >>>> As far as I can tell, the next steps would be: > >>>> > >>>> 1) Release flink-shaded with upgraded Jackson > >>>> 2a) Bump the flink-shaded version by default in master > >>>> 2b) Create opt-in profiles for 1.8 and 1.9 (the opt-in profiles > >>>> should also > >>>> cover the upgrade to the most recent flink-shaded version) > >>>> > >>>> @Chesnay: is this a correct summary? > >>>> > >>>> Note this would block the 1.8.3 release on step 1. As an upside, we > >>>> might > >>>> get some additional feedback until the 1.10 release with these > >>>> profiles in > >>>> case users make use of them with 1.8/1.9. > >>>> > >>>> – Ufuk > >>>> > >>>> On Fri, Nov 15, 2019 at 12:08 PM Chesnay Schepler <ches...@apache.org > > > >>>> wrote: > >>>>> The opt-in approach would only be used for 1.8.3 / 1.9.2; on master > >>>>> (and > >>>>> thus starting from 1.10.0) it's not opt-in. > >>>>> > >>>>> I have only proposed it as an opt-in because a) we usually do not > bump > >>>>> dependencies in bugfix releases and b) it's a short-term change that > we > >>>>> aren't allowing to mature properly. > >>>>> In contrast, the 1.10 release is significantly further away, hence no > >>>>> opt-in. > >>>>> > >>>>> Hence, I'm not concerned about such kind of ugprades being more > common > >>>>> in the future. > >>>>> > >>>>> We can certainly support every jackson version that fixes these > >>>>> vulnerabilities; individual modules can always use a different > version > >>>>> (that hopefully includes the fixes). > >>>>> Ideally of course we'd only be using 1 version, but that may or may > not > >>>>> be feasible. > >>>>> > >>>>> On 15/11/2019 04:07, Hequn Cheng wrote: > >>>>>> Hi Chesnay, > >>>>>> > >>>>>> Great to hear that jackson-2.10.1 works well on master. Really a > good > >>>> job! > >>>>>> - Whether backport this change to 1.8/1.9 > >>>>>> I had taken a quick look at the security vulnerabilities, some of > them > >>>>>> seem can lead to high-security problems, thus from my point of view, > >>>>>> I'm in favor of adding the fix into 1.9/1.8. However, I would like > to > >>>>>> trust your judgment as you are more professional at this problem. > >>>>>> > >>>>>> - How to port this change to 1.8/1.9 > >>>>>> I think providing an opt-in upgrade is a good idea. Another question > >>>>>> here is whether do we plan to support multi jackson versions that > have > >>>>>> eliminated the security vulnerabilities. If we only plan to support > >>>>>> 2.10.1, I would like to make it a non-opt-in upgrade. As an option, > >>>>>> users can downgrade the flink version if meet problems using the new > >>>>>> version. Of course, we will try our best to make the new release out > >>>>>> of question. > >>>>>> Another concern of making it an opt-in upgrade is, it will make our > >>>>>> build unlikely convergence as more and more build options will be > >>>>>> added when we upgrade a commonly used lib like this one. > >>>>>> > >>>>>> What do you think? > >>>>>> > >>>>>> Best, Hequn > >>>>>> > >>>>>> On Thu, Nov 14, 2019 at 6:00 PM Chesnay Schepler < > ches...@apache.org > >>>>>> <mailto:ches...@apache.org>> wrote: > >>>>>> > >>>>>> So here's the state of things: > >>>>>> > >>>>>> > >>>>>> The master of flink-shaded now uses jackson 2.10.1, which > >>>>>> eliminates a whole category of security vulnerabilities. > >>>>>> The flink master works perfectly fine with that version; 1.9 > will > >>>>>> likely do so too and 1.8 would require a minor adjustment. > >>>>>> > >>>>>> Hence, there may be value in first doing a flink-shaded > >>>>>> release so > >>>>>> we can eliminate these vulnerabilities in 1.8.3 and 1.9.2 . > >>>>>> > >>>>>> > >>>>>> As for other jackson dependencies (coming from calcite, kafka, > >>>>>> kinesis), I ran the unit and end-to-end tests of master > yesterday > >>>>>> will /all /jackson dependencies set to 2.10.1, and they > passed. I > >>>>>> will open a PR soon-ish for making this change on master. > >>>>>> > >>>>>> The question now is whether we want to backport this change to > >>>>>> 1.8/1.9 . > >>>>>> Some code paths /may /not be covered by our tests, and > transitive > >>>>>> jackson users /might /run into issues. > >>>>>> Alternatively, we could set this up as an opt-in upgrade, by > >>>>>> adding a separate profile that bumps the versions. This would > >>>>>> present users/providers who are concerned about the > >>>>>> vulnerabilities an easy workaround, at the risk of /some > /things > >>>>>> /maybe /not working. > >>>>>> > >>>>>> On 14/11/2019 03:16, Hequn Cheng wrote: > >>>>>>> Hi Chesnay, Jincheng > >>>>>>> > >>>>>>> Sure, I think it's good to have these fixes. > >>>>>>> Thanks a lot for providing the information about the security > >>>>>>> vulnerabilities! @Chesnay > >>>>>>> > >>>>>>> Best, Hequn > >>>>>>> > >>>>>>> On Thu, Nov 14, 2019 at 10:07 AM jincheng sun< > >>>> sunjincheng...@gmail.com> <mailto:sunjincheng...@gmail.com> > >>>>>>> wrote: > >>>>>>> > >>>>>>>> +1 for try to eliminate the security vulnerabilities. Great > >>>> thanks for > >>>>>>>> doing this important work, Chesnay! > >>>>>>>> What do you think Hequn ? > >>>>>>>> > >>>>>>>> Best, > >>>>>>>> Jincheng > >>>>>>>> > >>>>>>>> Chesnay Schepler<ches...@apache.org> > >>>>>>>> <mailto:ches...@apache.org> > >>>> 于2019年11月13日周三 下午5:17写道： > >>>>>>>>> It would be great if you could give me a day or 2 to check > how > >>>> easy it > >>>>>>>>> would be to bump the various jackson dependencies to > >>>>>>>>> eliminate a > >>>> few > >>>>>>>>> security vulnerabilities. > >>>>>>>>> > >>>>>>>>> On 09/11/2019 05:10, jincheng sun wrote: > >>>>>>>>>> Hi Flink devs, > >>>>>>>>>> > >>>>>>>>>> It has been more than 2 months since the 1.8.2 released. > So, > >>>> What do > >>>>>>>> you > >>>>>>>>>> think about releasing Flink 1.8.3 soon? > >>>>>>>>>> > >>>>>>>>>> We already have many important bug fixes in the > release-1.8 > >>>> branch (29 > >>>>>>>>>> resolved issues). > >>>>>>>>>> > >>>>>>>>>> Most notable fixes are: > >>>>>>>>>> > >>>>>>>>>> - FLINK-14010 Dispatcher & JobManagers don't give up > >>>>>>>>>> leadership > >>>> when AM > >>>>>>>>> is > >>>>>>>>>> shut down > >>>>>>>>>> - FLINK-14315 NPE with JobMaster.disconnectTaskManager > >>>>>>>>>> - FLINK-12848 Method equals() in RowTypeInfo should > consider > >>>>>>>> fieldsNames > >>>>>>>>>> - FLINK-12342 Yarn Resource Manager Acquires Too Many > >>>>>>>>>> Containers > >>>>>>>>>> - FLINK-14589 Redundant slot requests with the same > >>>> AllocationID leads > >>>>>>>> to > >>>>>>>>>> inconsistent slot table > >>>>>>>>>> > >>>>>>>>>> Furthermore, the following critical issues is in progress, > >>>> maybe we can > >>>>>>>>>> wait for it if it is not too much effort. > >>>>>>>>>> > >>>>>>>>>> - FLINK-13184 Starting a TaskExecutor blocks the > >>>> YarnResourceManager's > >>>>>>>>> main > >>>>>>>>>> thread > >>>>>>>>>> > >>>>>>>>>> Please let me know what you think? > >>>>>>>>>> > >>>>>>>>>> Best, > >>>>>>>>>> Jincheng > >>>>>>>>>> > >>> > >> > >
