@Hequn: flink-shaded:9.0 is available in Maven central now. I think you can go ahead and create the first RC. :-)
On Mon, Nov 25, 2019 at 7:47 AM Zhu Zhu <reed...@gmail.com> wrote: > Hi Hequn, > > Looks we are not able to merge fix of FLINK-14735 to 1.8 very soon. > Given that this fix is for batch job only and batch is not very good in > 1.8, I think it is a not blocker of release 1.8.3. > So just don't be blocked by it and feel free to cut the RC when other > blocking issues are resolved. > > Thanks, > Zhu Zhu > > Hequn Cheng <chenghe...@gmail.com> 于2019年11月23日周六 下午9:08写道： > > > Hi Zhu Zhu, > > > > Thanks a lot for letting us know! > > We can't cut the first RC right now due to the wait of the flink-shade > > release, so go ahead. > > > > Theoretically, we will cut the first RC of 1.8.3 and vote for it once the > > release of flink-shade is done, > > but I will try my best to have it in 1.8.3. Hope we can get it on board > on > > time. :) > > > > Best, Hequn > > > > On Sat, Nov 23, 2019 at 10:40 AM Zhu Zhu <reed...@gmail.com> wrote: > > > >> Hi Jincheng & Hequn > >> > >> Thanks for driving the releasing of 1.8.3. > >> > >> I am now working on FLINK-14735. The fix avoids duplicated input > >> checking when scheduling ALL-to-ALL > >> connected downstream consumers with ALL input constraints. The > duplicated > >> checking can cause severe > >> performance issues for large scale jobs. So I hope the fix could be > >> released with 1.8.3. > >> > >> The fix is already merged into master, and is now in the process of > >> backporting to 1.8. > >> > >> Thanks, > >> Zhu Zhu > >> > >> Ufuk Celebi <u...@apache.org> 于2019年11月15日周五 下午11:54写道： > >> > >>> Thanks Chesnay. > >>> > >>> I'm also +1 to release 1.8.3 asap without the changes for the Jackson > >>> version bump and leave those for a future release. Realistically, the > >>> flink-shaded release will take until mid next week or end of next week. > >>> But > >>> please correct me if you think that it should not take that long or > it's > >>> OK > >>> to block the 1.8.3 release on the flink-shaded release. > >>> > >>> – Ufuk > >>> > >>> > >>> On Fri, Nov 15, 2019 at 2:27 PM Chesnay Schepler <ches...@apache.org> > >>> wrote: > >>> > >>> > I've kicked off a discussion about the next flink-shaded release, and > >>> > have opened PRs for adding the opt-in profile to 1.8/1.9. > >>> > > >>> > On 15/11/2019 13:54, Hequn Cheng wrote: > >>> > > That's great, thank you very much! Ideally, we can kick off the > >>> release > >>> > > vote for the first RC of 1.8.3 within next week. :) > >>> > > > >>> > > On Fri, Nov 15, 2019 at 8:47 PM Chesnay Schepler < > ches...@apache.org > >>> > > >>> > wrote: > >>> > > > >>> > >> I'm not aware of any more planned changes to flink-shaded; so we > >>> could > >>> > >> start the release right away. > >>> > >> > >>> > >> On 15/11/2019 13:44, Hequn Cheng wrote: > >>> > >>> Hi, > >>> > >>> > >>> > >>> @Chesnay Thanks a lot for the explanation. +1 to the opt-in > >>> approach > >>> > for > >>> > >>> 1.8/1.9. > >>> > >>> @Ufuk Thank you for the nice summary. > >>> > >>> > >>> > >>> Looks good so far except that we need to postpone 1.8.3 a bit to > >>> first > >>> > >> do a > >>> > >>> flink-shaded release. > >>> > >>> BTW, @chesnay when would we plan to release the flink-shaded with > >>> > >> upgraded > >>> > >>> Jackson? > >>> > >>> > >>> > >>> Best, Hequn > >>> > >>> > >>> > >>> On Fri, Nov 15, 2019 at 7:43 PM Chesnay Schepler < > >>> ches...@apache.org> > >>> > >> wrote: > >>> > >>>> One small modification: the flink-shaded upgrade does not have > to > >>> be > >>> > >>>> part of the profile; since it is only intended for internal use > >>> anyway > >>> > >>>> (and thus has limited exposure) we can be pretty sure this > doesn't > >>> > break > >>> > >>>> anything. > >>> > >>>> > >>> > >>>> On 15/11/2019 12:23, Chesnay Schepler wrote: > >>> > >>>>> Ufuk's summary is correct. > >>> > >>>>> > >>> > >>>>> There's a slight caveat in that we'd also have to bump the > >>> > >>>>> shade-plugin to 3.1.1 since it otherwise fails on jackson, > >>> > >>>>> but I have no concerns about this change. > >>> > >>>>> > >>> > >>>>> On 15/11/2019 12:19, Ufuk Celebi wrote: > >>> > >>>>>> The opt-in approach seems reasonable to me. +1 to include the > >>> > >>>>>> profiles in > >>> > >>>>>> 1.8 and 1.9 without changing the default versions (including > the > >>> > >> default > >>> > >>>>>> version of flink-shaded). > >>> > >>>>>> > >>> > >>>>>> As far as I can tell, the next steps would be: > >>> > >>>>>> > >>> > >>>>>> 1) Release flink-shaded with upgraded Jackson > >>> > >>>>>> 2a) Bump the flink-shaded version by default in master > >>> > >>>>>> 2b) Create opt-in profiles for 1.8 and 1.9 (the opt-in > profiles > >>> > >>>>>> should also > >>> > >>>>>> cover the upgrade to the most recent flink-shaded version) > >>> > >>>>>> > >>> > >>>>>> @Chesnay: is this a correct summary? > >>> > >>>>>> > >>> > >>>>>> Note this would block the 1.8.3 release on step 1. As an > >>> upside, we > >>> > >>>>>> might > >>> > >>>>>> get some additional feedback until the 1.10 release with these > >>> > >>>>>> profiles in > >>> > >>>>>> case users make use of them with 1.8/1.9. > >>> > >>>>>> > >>> > >>>>>> – Ufuk > >>> > >>>>>> > >>> > >>>>>> On Fri, Nov 15, 2019 at 12:08 PM Chesnay Schepler < > >>> > ches...@apache.org > >>> > >>>>>> wrote: > >>> > >>>>>>> The opt-in approach would only be used for 1.8.3 / 1.9.2; on > >>> master > >>> > >>>>>>> (and > >>> > >>>>>>> thus starting from 1.10.0) it's not opt-in. > >>> > >>>>>>> > >>> > >>>>>>> I have only proposed it as an opt-in because a) we usually do > >>> not > >>> > >> bump > >>> > >>>>>>> dependencies in bugfix releases and b) it's a short-term > change > >>> > that > >>> > >> we > >>> > >>>>>>> aren't allowing to mature properly. > >>> > >>>>>>> In contrast, the 1.10 release is significantly further away, > >>> hence > >>> > no > >>> > >>>>>>> opt-in. > >>> > >>>>>>> > >>> > >>>>>>> Hence, I'm not concerned about such kind of ugprades being > more > >>> > >> common > >>> > >>>>>>> in the future. > >>> > >>>>>>> > >>> > >>>>>>> We can certainly support every jackson version that fixes > these > >>> > >>>>>>> vulnerabilities; individual modules can always use a > different > >>> > >> version > >>> > >>>>>>> (that hopefully includes the fixes). > >>> > >>>>>>> Ideally of course we'd only be using 1 version, but that may > >>> or may > >>> > >> not > >>> > >>>>>>> be feasible. > >>> > >>>>>>> > >>> > >>>>>>> On 15/11/2019 04:07, Hequn Cheng wrote: > >>> > >>>>>>>> Hi Chesnay, > >>> > >>>>>>>> > >>> > >>>>>>>> Great to hear that jackson-2.10.1 works well on master. > >>> Really a > >>> > >> good > >>> > >>>>>> job! > >>> > >>>>>>>> - Whether backport this change to 1.8/1.9 > >>> > >>>>>>>> I had taken a quick look at the security vulnerabilities, > >>> some of > >>> > >> them > >>> > >>>>>>>> seem can lead to high-security problems, thus from my point > of > >>> > view, > >>> > >>>>>>>> I'm in favor of adding the fix into 1.9/1.8. However, I > would > >>> like > >>> > >> to > >>> > >>>>>>>> trust your judgment as you are more professional at this > >>> problem. > >>> > >>>>>>>> > >>> > >>>>>>>> - How to port this change to 1.8/1.9 > >>> > >>>>>>>> I think providing an opt-in upgrade is a good idea. Another > >>> > question > >>> > >>>>>>>> here is whether do we plan to support multi jackson versions > >>> that > >>> > >> have > >>> > >>>>>>>> eliminated the security vulnerabilities. If we only plan to > >>> > support > >>> > >>>>>>>> 2.10.1, I would like to make it a non-opt-in upgrade. As an > >>> > option, > >>> > >>>>>>>> users can downgrade the flink version if meet problems using > >>> the > >>> > new > >>> > >>>>>>>> version. Of course, we will try our best to make the new > >>> release > >>> > out > >>> > >>>>>>>> of question. > >>> > >>>>>>>> Another concern of making it an opt-in upgrade is, it will > >>> make > >>> > our > >>> > >>>>>>>> build unlikely convergence as more and more build options > >>> will be > >>> > >>>>>>>> added when we upgrade a commonly used lib like this one. > >>> > >>>>>>>> > >>> > >>>>>>>> What do you think? > >>> > >>>>>>>> > >>> > >>>>>>>> Best, Hequn > >>> > >>>>>>>> > >>> > >>>>>>>> On Thu, Nov 14, 2019 at 6:00 PM Chesnay Schepler < > >>> > >> ches...@apache.org > >>> > >>>>>>>> <mailto:ches...@apache.org>> wrote: > >>> > >>>>>>>> > >>> > >>>>>>>> So here's the state of things: > >>> > >>>>>>>> > >>> > >>>>>>>> > >>> > >>>>>>>> The master of flink-shaded now uses jackson 2.10.1, > >>> which > >>> > >>>>>>>> eliminates a whole category of security > >>> vulnerabilities. > >>> > >>>>>>>> The flink master works perfectly fine with that > >>> version; > >>> > 1.9 > >>> > >> will > >>> > >>>>>>>> likely do so too and 1.8 would require a minor > >>> adjustment. > >>> > >>>>>>>> > >>> > >>>>>>>> Hence, there may be value in first doing a > flink-shaded > >>> > >>>>>>>> release so > >>> > >>>>>>>> we can eliminate these vulnerabilities in 1.8.3 and > >>> 1.9.2 . > >>> > >>>>>>>> > >>> > >>>>>>>> > >>> > >>>>>>>> As for other jackson dependencies (coming from > calcite, > >>> > kafka, > >>> > >>>>>>>> kinesis), I ran the unit and end-to-end tests of > master > >>> > >> yesterday > >>> > >>>>>>>> will /all /jackson dependencies set to 2.10.1, and > they > >>> > >> passed. I > >>> > >>>>>>>> will open a PR soon-ish for making this change on > >>> master. > >>> > >>>>>>>> > >>> > >>>>>>>> The question now is whether we want to backport this > >>> > change to > >>> > >>>>>>>> 1.8/1.9 . > >>> > >>>>>>>> Some code paths /may /not be covered by our tests, > and > >>> > >> transitive > >>> > >>>>>>>> jackson users /might /run into issues. > >>> > >>>>>>>> Alternatively, we could set this up as an opt-in > >>> upgrade, > >>> > by > >>> > >>>>>>>> adding a separate profile that bumps the versions. > This > >>> > would > >>> > >>>>>>>> present users/providers who are concerned about the > >>> > >>>>>>>> vulnerabilities an easy workaround, at the risk of > >>> /some > >>> > >> /things > >>> > >>>>>>>> /maybe /not working. > >>> > >>>>>>>> > >>> > >>>>>>>> On 14/11/2019 03:16, Hequn Cheng wrote: > >>> > >>>>>>>>> Hi Chesnay, Jincheng > >>> > >>>>>>>>> > >>> > >>>>>>>>> Sure, I think it's good to have these fixes. > >>> > >>>>>>>>> Thanks a lot for providing the information about the > >>> > security > >>> > >>>>>>>>> vulnerabilities! @Chesnay > >>> > >>>>>>>>> > >>> > >>>>>>>>> Best, Hequn > >>> > >>>>>>>>> > >>> > >>>>>>>>> On Thu, Nov 14, 2019 at 10:07 AM jincheng sun< > >>> > >>>>>> sunjincheng...@gmail.com> <mailto:sunjincheng...@gmail.com> > >>> > >>>>>>>>> wrote: > >>> > >>>>>>>>> > >>> > >>>>>>>>>> +1 for try to eliminate the security > vulnerabilities. > >>> > Great > >>> > >>>>>> thanks for > >>> > >>>>>>>>>> doing this important work, Chesnay! > >>> > >>>>>>>>>> What do you think Hequn ? > >>> > >>>>>>>>>> > >>> > >>>>>>>>>> Best, > >>> > >>>>>>>>>> Jincheng > >>> > >>>>>>>>>> > >>> > >>>>>>>>>> Chesnay Schepler<ches...@apache.org> > >>> > >>>>>>>>>> <mailto:ches...@apache.org> > >>> > >>>>>> 于2019年11月13日周三 下午5:17写道： > >>> > >>>>>>>>>>> It would be great if you could give me a day or 2 > to > >>> > check > >>> > >> how > >>> > >>>>>> easy it > >>> > >>>>>>>>>>> would be to bump the various jackson dependencies > to > >>> > >>>>>>>>>>> eliminate a > >>> > >>>>>> few > >>> > >>>>>>>>>>> security vulnerabilities. > >>> > >>>>>>>>>>> > >>> > >>>>>>>>>>> On 09/11/2019 05:10, jincheng sun wrote: > >>> > >>>>>>>>>>>> Hi Flink devs, > >>> > >>>>>>>>>>>> > >>> > >>>>>>>>>>>> It has been more than 2 months since the 1.8.2 > >>> > released. > >>> > >> So, > >>> > >>>>>> What do > >>> > >>>>>>>>>> you > >>> > >>>>>>>>>>>> think about releasing Flink 1.8.3 soon? > >>> > >>>>>>>>>>>> > >>> > >>>>>>>>>>>> We already have many important bug fixes in the > >>> > >> release-1.8 > >>> > >>>>>> branch (29 > >>> > >>>>>>>>>>>> resolved issues). > >>> > >>>>>>>>>>>> > >>> > >>>>>>>>>>>> Most notable fixes are: > >>> > >>>>>>>>>>>> > >>> > >>>>>>>>>>>> - FLINK-14010 Dispatcher & JobManagers don't give > >>> up > >>> > >>>>>>>>>>>> leadership > >>> > >>>>>> when AM > >>> > >>>>>>>>>>> is > >>> > >>>>>>>>>>>> shut down > >>> > >>>>>>>>>>>> - FLINK-14315 NPE with > >>> JobMaster.disconnectTaskManager > >>> > >>>>>>>>>>>> - FLINK-12848 Method equals() in RowTypeInfo > should > >>> > >> consider > >>> > >>>>>>>>>> fieldsNames > >>> > >>>>>>>>>>>> - FLINK-12342 Yarn Resource Manager Acquires Too > >>> Many > >>> > >>>>>>>>>>>> Containers > >>> > >>>>>>>>>>>> - FLINK-14589 Redundant slot requests with the > same > >>> > >>>>>> AllocationID leads > >>> > >>>>>>>>>> to > >>> > >>>>>>>>>>>> inconsistent slot table > >>> > >>>>>>>>>>>> > >>> > >>>>>>>>>>>> Furthermore, the following critical issues is in > >>> > progress, > >>> > >>>>>> maybe we can > >>> > >>>>>>>>>>>> wait for it if it is not too much effort. > >>> > >>>>>>>>>>>> > >>> > >>>>>>>>>>>> - FLINK-13184 Starting a TaskExecutor blocks the > >>> > >>>>>> YarnResourceManager's > >>> > >>>>>>>>>>> main > >>> > >>>>>>>>>>>> thread > >>> > >>>>>>>>>>>> > >>> > >>>>>>>>>>>> Please let me know what you think? > >>> > >>>>>>>>>>>> > >>> > >>>>>>>>>>>> Best, > >>> > >>>>>>>>>>>> Jincheng > >>> > >>>>>>>>>>>> > >>> > >> > >>> > > >>> > > >>> > >> >
