Thanks Chesnay. I'm also +1 to release 1.8.3 asap without the changes for the Jackson version bump and leave those for a future release. Realistically, the flink-shaded release will take until mid next week or end of next week. But please correct me if you think that it should not take that long or it's OK to block the 1.8.3 release on the flink-shaded release.
– Ufuk On Fri, Nov 15, 2019 at 2:27 PM Chesnay Schepler <ches...@apache.org> wrote: > I've kicked off a discussion about the next flink-shaded release, and > have opened PRs for adding the opt-in profile to 1.8/1.9. > > On 15/11/2019 13:54, Hequn Cheng wrote: > > That's great, thank you very much! Ideally, we can kick off the release > > vote for the first RC of 1.8.3 within next week. :) > > > > On Fri, Nov 15, 2019 at 8:47 PM Chesnay Schepler <ches...@apache.org> > wrote: > > > >> I'm not aware of any more planned changes to flink-shaded; so we could > >> start the release right away. > >> > >> On 15/11/2019 13:44, Hequn Cheng wrote: > >>> Hi, > >>> > >>> @Chesnay Thanks a lot for the explanation. +1 to the opt-in approach > for > >>> 1.8/1.9. > >>> @Ufuk Thank you for the nice summary. > >>> > >>> Looks good so far except that we need to postpone 1.8.3 a bit to first > >> do a > >>> flink-shaded release. > >>> BTW, @chesnay when would we plan to release the flink-shaded with > >> upgraded > >>> Jackson? > >>> > >>> Best, Hequn > >>> > >>> On Fri, Nov 15, 2019 at 7:43 PM Chesnay Schepler <ches...@apache.org> > >> wrote: > >>>> One small modification: the flink-shaded upgrade does not have to be > >>>> part of the profile; since it is only intended for internal use anyway > >>>> (and thus has limited exposure) we can be pretty sure this doesn't > break > >>>> anything. > >>>> > >>>> On 15/11/2019 12:23, Chesnay Schepler wrote: > >>>>> Ufuk's summary is correct. > >>>>> > >>>>> There's a slight caveat in that we'd also have to bump the > >>>>> shade-plugin to 3.1.1 since it otherwise fails on jackson, > >>>>> but I have no concerns about this change. > >>>>> > >>>>> On 15/11/2019 12:19, Ufuk Celebi wrote: > >>>>>> The opt-in approach seems reasonable to me. +1 to include the > >>>>>> profiles in > >>>>>> 1.8 and 1.9 without changing the default versions (including the > >> default > >>>>>> version of flink-shaded). > >>>>>> > >>>>>> As far as I can tell, the next steps would be: > >>>>>> > >>>>>> 1) Release flink-shaded with upgraded Jackson > >>>>>> 2a) Bump the flink-shaded version by default in master > >>>>>> 2b) Create opt-in profiles for 1.8 and 1.9 (the opt-in profiles > >>>>>> should also > >>>>>> cover the upgrade to the most recent flink-shaded version) > >>>>>> > >>>>>> @Chesnay: is this a correct summary? > >>>>>> > >>>>>> Note this would block the 1.8.3 release on step 1. As an upside, we > >>>>>> might > >>>>>> get some additional feedback until the 1.10 release with these > >>>>>> profiles in > >>>>>> case users make use of them with 1.8/1.9. > >>>>>> > >>>>>> – Ufuk > >>>>>> > >>>>>> On Fri, Nov 15, 2019 at 12:08 PM Chesnay Schepler < > ches...@apache.org > >>>>>> wrote: > >>>>>>> The opt-in approach would only be used for 1.8.3 / 1.9.2; on master > >>>>>>> (and > >>>>>>> thus starting from 1.10.0) it's not opt-in. > >>>>>>> > >>>>>>> I have only proposed it as an opt-in because a) we usually do not > >> bump > >>>>>>> dependencies in bugfix releases and b) it's a short-term change > that > >> we > >>>>>>> aren't allowing to mature properly. > >>>>>>> In contrast, the 1.10 release is significantly further away, hence > no > >>>>>>> opt-in. > >>>>>>> > >>>>>>> Hence, I'm not concerned about such kind of ugprades being more > >> common > >>>>>>> in the future. > >>>>>>> > >>>>>>> We can certainly support every jackson version that fixes these > >>>>>>> vulnerabilities; individual modules can always use a different > >> version > >>>>>>> (that hopefully includes the fixes). > >>>>>>> Ideally of course we'd only be using 1 version, but that may or may > >> not > >>>>>>> be feasible. > >>>>>>> > >>>>>>> On 15/11/2019 04:07, Hequn Cheng wrote: > >>>>>>>> Hi Chesnay, > >>>>>>>> > >>>>>>>> Great to hear that jackson-2.10.1 works well on master. Really a > >> good > >>>>>> job! > >>>>>>>> - Whether backport this change to 1.8/1.9 > >>>>>>>> I had taken a quick look at the security vulnerabilities, some of > >> them > >>>>>>>> seem can lead to high-security problems, thus from my point of > view, > >>>>>>>> I'm in favor of adding the fix into 1.9/1.8. However, I would like > >> to > >>>>>>>> trust your judgment as you are more professional at this problem. > >>>>>>>> > >>>>>>>> - How to port this change to 1.8/1.9 > >>>>>>>> I think providing an opt-in upgrade is a good idea. Another > question > >>>>>>>> here is whether do we plan to support multi jackson versions that > >> have > >>>>>>>> eliminated the security vulnerabilities. If we only plan to > support > >>>>>>>> 2.10.1, I would like to make it a non-opt-in upgrade. As an > option, > >>>>>>>> users can downgrade the flink version if meet problems using the > new > >>>>>>>> version. Of course, we will try our best to make the new release > out > >>>>>>>> of question. > >>>>>>>> Another concern of making it an opt-in upgrade is, it will make > our > >>>>>>>> build unlikely convergence as more and more build options will be > >>>>>>>> added when we upgrade a commonly used lib like this one. > >>>>>>>> > >>>>>>>> What do you think? > >>>>>>>> > >>>>>>>> Best, Hequn > >>>>>>>> > >>>>>>>> On Thu, Nov 14, 2019 at 6:00 PM Chesnay Schepler < > >> ches...@apache.org > >>>>>>>> <mailto:ches...@apache.org>> wrote: > >>>>>>>> > >>>>>>>> So here's the state of things: > >>>>>>>> > >>>>>>>> > >>>>>>>> The master of flink-shaded now uses jackson 2.10.1, which > >>>>>>>> eliminates a whole category of security vulnerabilities. > >>>>>>>> The flink master works perfectly fine with that version; > 1.9 > >> will > >>>>>>>> likely do so too and 1.8 would require a minor adjustment. > >>>>>>>> > >>>>>>>> Hence, there may be value in first doing a flink-shaded > >>>>>>>> release so > >>>>>>>> we can eliminate these vulnerabilities in 1.8.3 and 1.9.2 . > >>>>>>>> > >>>>>>>> > >>>>>>>> As for other jackson dependencies (coming from calcite, > kafka, > >>>>>>>> kinesis), I ran the unit and end-to-end tests of master > >> yesterday > >>>>>>>> will /all /jackson dependencies set to 2.10.1, and they > >> passed. I > >>>>>>>> will open a PR soon-ish for making this change on master. > >>>>>>>> > >>>>>>>> The question now is whether we want to backport this > change to > >>>>>>>> 1.8/1.9 . > >>>>>>>> Some code paths /may /not be covered by our tests, and > >> transitive > >>>>>>>> jackson users /might /run into issues. > >>>>>>>> Alternatively, we could set this up as an opt-in upgrade, > by > >>>>>>>> adding a separate profile that bumps the versions. This > would > >>>>>>>> present users/providers who are concerned about the > >>>>>>>> vulnerabilities an easy workaround, at the risk of /some > >> /things > >>>>>>>> /maybe /not working. > >>>>>>>> > >>>>>>>> On 14/11/2019 03:16, Hequn Cheng wrote: > >>>>>>>>> Hi Chesnay, Jincheng > >>>>>>>>> > >>>>>>>>> Sure, I think it's good to have these fixes. > >>>>>>>>> Thanks a lot for providing the information about the > security > >>>>>>>>> vulnerabilities! @Chesnay > >>>>>>>>> > >>>>>>>>> Best, Hequn > >>>>>>>>> > >>>>>>>>> On Thu, Nov 14, 2019 at 10:07 AM jincheng sun< > >>>>>> sunjincheng...@gmail.com> <mailto:sunjincheng...@gmail.com> > >>>>>>>>> wrote: > >>>>>>>>> > >>>>>>>>>> +1 for try to eliminate the security vulnerabilities. > Great > >>>>>> thanks for > >>>>>>>>>> doing this important work, Chesnay! > >>>>>>>>>> What do you think Hequn ? > >>>>>>>>>> > >>>>>>>>>> Best, > >>>>>>>>>> Jincheng > >>>>>>>>>> > >>>>>>>>>> Chesnay Schepler<ches...@apache.org> > >>>>>>>>>> <mailto:ches...@apache.org> > >>>>>> 于2019年11月13日周三 下午5:17写道： > >>>>>>>>>>> It would be great if you could give me a day or 2 to > check > >> how > >>>>>> easy it > >>>>>>>>>>> would be to bump the various jackson dependencies to > >>>>>>>>>>> eliminate a > >>>>>> few > >>>>>>>>>>> security vulnerabilities. > >>>>>>>>>>> > >>>>>>>>>>> On 09/11/2019 05:10, jincheng sun wrote: > >>>>>>>>>>>> Hi Flink devs, > >>>>>>>>>>>> > >>>>>>>>>>>> It has been more than 2 months since the 1.8.2 > released. > >> So, > >>>>>> What do > >>>>>>>>>> you > >>>>>>>>>>>> think about releasing Flink 1.8.3 soon? > >>>>>>>>>>>> > >>>>>>>>>>>> We already have many important bug fixes in the > >> release-1.8 > >>>>>> branch (29 > >>>>>>>>>>>> resolved issues). > >>>>>>>>>>>> > >>>>>>>>>>>> Most notable fixes are: > >>>>>>>>>>>> > >>>>>>>>>>>> - FLINK-14010 Dispatcher & JobManagers don't give up > >>>>>>>>>>>> leadership > >>>>>> when AM > >>>>>>>>>>> is > >>>>>>>>>>>> shut down > >>>>>>>>>>>> - FLINK-14315 NPE with JobMaster.disconnectTaskManager > >>>>>>>>>>>> - FLINK-12848 Method equals() in RowTypeInfo should > >> consider > >>>>>>>>>> fieldsNames > >>>>>>>>>>>> - FLINK-12342 Yarn Resource Manager Acquires Too Many > >>>>>>>>>>>> Containers > >>>>>>>>>>>> - FLINK-14589 Redundant slot requests with the same > >>>>>> AllocationID leads > >>>>>>>>>> to > >>>>>>>>>>>> inconsistent slot table > >>>>>>>>>>>> > >>>>>>>>>>>> Furthermore, the following critical issues is in > progress, > >>>>>> maybe we can > >>>>>>>>>>>> wait for it if it is not too much effort. > >>>>>>>>>>>> > >>>>>>>>>>>> - FLINK-13184 Starting a TaskExecutor blocks the > >>>>>> YarnResourceManager's > >>>>>>>>>>> main > >>>>>>>>>>>> thread > >>>>>>>>>>>> > >>>>>>>>>>>> Please let me know what you think? > >>>>>>>>>>>> > >>>>>>>>>>>> Best, > >>>>>>>>>>>> Jincheng > >>>>>>>>>>>> > >> > >
