Hi Zhu Zhu, Thanks a lot for letting us know! We can't cut the first RC right now due to the wait of the flink-shade release, so go ahead.
Theoretically, we will cut the first RC of 1.8.3 and vote for it once the release of flink-shade is done, but I will try my best to have it in 1.8.3. Hope we can get it on board on time. :) Best, Hequn On Sat, Nov 23, 2019 at 10:40 AM Zhu Zhu <reed...@gmail.com> wrote: > Hi Jincheng & Hequn > > Thanks for driving the releasing of 1.8.3. > > I am now working on FLINK-14735. The fix avoids duplicated input > checking when scheduling ALL-to-ALL > connected downstream consumers with ALL input constraints. The duplicated > checking can cause severe > performance issues for large scale jobs. So I hope the fix could be > released with 1.8.3. > > The fix is already merged into master, and is now in the process of > backporting to 1.8. > > Thanks, > Zhu Zhu > > Ufuk Celebi <u...@apache.org> 于2019年11月15日周五 下午11:54写道： > >> Thanks Chesnay. >> >> I'm also +1 to release 1.8.3 asap without the changes for the Jackson >> version bump and leave those for a future release. Realistically, the >> flink-shaded release will take until mid next week or end of next week. >> But >> please correct me if you think that it should not take that long or it's >> OK >> to block the 1.8.3 release on the flink-shaded release. >> >> – Ufuk >> >> >> On Fri, Nov 15, 2019 at 2:27 PM Chesnay Schepler <ches...@apache.org> >> wrote: >> >> > I've kicked off a discussion about the next flink-shaded release, and >> > have opened PRs for adding the opt-in profile to 1.8/1.9. >> > >> > On 15/11/2019 13:54, Hequn Cheng wrote: >> > > That's great, thank you very much! Ideally, we can kick off the >> release >> > > vote for the first RC of 1.8.3 within next week. :) >> > > >> > > On Fri, Nov 15, 2019 at 8:47 PM Chesnay Schepler <ches...@apache.org> >> > wrote: >> > > >> > >> I'm not aware of any more planned changes to flink-shaded; so we >> could >> > >> start the release right away. >> > >> >> > >> On 15/11/2019 13:44, Hequn Cheng wrote: >> > >>> Hi, >> > >>> >> > >>> @Chesnay Thanks a lot for the explanation. +1 to the opt-in approach >> > for >> > >>> 1.8/1.9. >> > >>> @Ufuk Thank you for the nice summary. >> > >>> >> > >>> Looks good so far except that we need to postpone 1.8.3 a bit to >> first >> > >> do a >> > >>> flink-shaded release. >> > >>> BTW, @chesnay when would we plan to release the flink-shaded with >> > >> upgraded >> > >>> Jackson? >> > >>> >> > >>> Best, Hequn >> > >>> >> > >>> On Fri, Nov 15, 2019 at 7:43 PM Chesnay Schepler < >> ches...@apache.org> >> > >> wrote: >> > >>>> One small modification: the flink-shaded upgrade does not have to >> be >> > >>>> part of the profile; since it is only intended for internal use >> anyway >> > >>>> (and thus has limited exposure) we can be pretty sure this doesn't >> > break >> > >>>> anything. >> > >>>> >> > >>>> On 15/11/2019 12:23, Chesnay Schepler wrote: >> > >>>>> Ufuk's summary is correct. >> > >>>>> >> > >>>>> There's a slight caveat in that we'd also have to bump the >> > >>>>> shade-plugin to 3.1.1 since it otherwise fails on jackson, >> > >>>>> but I have no concerns about this change. >> > >>>>> >> > >>>>> On 15/11/2019 12:19, Ufuk Celebi wrote: >> > >>>>>> The opt-in approach seems reasonable to me. +1 to include the >> > >>>>>> profiles in >> > >>>>>> 1.8 and 1.9 without changing the default versions (including the >> > >> default >> > >>>>>> version of flink-shaded). >> > >>>>>> >> > >>>>>> As far as I can tell, the next steps would be: >> > >>>>>> >> > >>>>>> 1) Release flink-shaded with upgraded Jackson >> > >>>>>> 2a) Bump the flink-shaded version by default in master >> > >>>>>> 2b) Create opt-in profiles for 1.8 and 1.9 (the opt-in profiles >> > >>>>>> should also >> > >>>>>> cover the upgrade to the most recent flink-shaded version) >> > >>>>>> >> > >>>>>> @Chesnay: is this a correct summary? >> > >>>>>> >> > >>>>>> Note this would block the 1.8.3 release on step 1. As an upside, >> we >> > >>>>>> might >> > >>>>>> get some additional feedback until the 1.10 release with these >> > >>>>>> profiles in >> > >>>>>> case users make use of them with 1.8/1.9. >> > >>>>>> >> > >>>>>> – Ufuk >> > >>>>>> >> > >>>>>> On Fri, Nov 15, 2019 at 12:08 PM Chesnay Schepler < >> > ches...@apache.org >> > >>>>>> wrote: >> > >>>>>>> The opt-in approach would only be used for 1.8.3 / 1.9.2; on >> master >> > >>>>>>> (and >> > >>>>>>> thus starting from 1.10.0) it's not opt-in. >> > >>>>>>> >> > >>>>>>> I have only proposed it as an opt-in because a) we usually do >> not >> > >> bump >> > >>>>>>> dependencies in bugfix releases and b) it's a short-term change >> > that >> > >> we >> > >>>>>>> aren't allowing to mature properly. >> > >>>>>>> In contrast, the 1.10 release is significantly further away, >> hence >> > no >> > >>>>>>> opt-in. >> > >>>>>>> >> > >>>>>>> Hence, I'm not concerned about such kind of ugprades being more >> > >> common >> > >>>>>>> in the future. >> > >>>>>>> >> > >>>>>>> We can certainly support every jackson version that fixes these >> > >>>>>>> vulnerabilities; individual modules can always use a different >> > >> version >> > >>>>>>> (that hopefully includes the fixes). >> > >>>>>>> Ideally of course we'd only be using 1 version, but that may or >> may >> > >> not >> > >>>>>>> be feasible. >> > >>>>>>> >> > >>>>>>> On 15/11/2019 04:07, Hequn Cheng wrote: >> > >>>>>>>> Hi Chesnay, >> > >>>>>>>> >> > >>>>>>>> Great to hear that jackson-2.10.1 works well on master. Really >> a >> > >> good >> > >>>>>> job! >> > >>>>>>>> - Whether backport this change to 1.8/1.9 >> > >>>>>>>> I had taken a quick look at the security vulnerabilities, some >> of >> > >> them >> > >>>>>>>> seem can lead to high-security problems, thus from my point of >> > view, >> > >>>>>>>> I'm in favor of adding the fix into 1.9/1.8. However, I would >> like >> > >> to >> > >>>>>>>> trust your judgment as you are more professional at this >> problem. >> > >>>>>>>> >> > >>>>>>>> - How to port this change to 1.8/1.9 >> > >>>>>>>> I think providing an opt-in upgrade is a good idea. Another >> > question >> > >>>>>>>> here is whether do we plan to support multi jackson versions >> that >> > >> have >> > >>>>>>>> eliminated the security vulnerabilities. If we only plan to >> > support >> > >>>>>>>> 2.10.1, I would like to make it a non-opt-in upgrade. As an >> > option, >> > >>>>>>>> users can downgrade the flink version if meet problems using >> the >> > new >> > >>>>>>>> version. Of course, we will try our best to make the new >> release >> > out >> > >>>>>>>> of question. >> > >>>>>>>> Another concern of making it an opt-in upgrade is, it will make >> > our >> > >>>>>>>> build unlikely convergence as more and more build options will >> be >> > >>>>>>>> added when we upgrade a commonly used lib like this one. >> > >>>>>>>> >> > >>>>>>>> What do you think? >> > >>>>>>>> >> > >>>>>>>> Best, Hequn >> > >>>>>>>> >> > >>>>>>>> On Thu, Nov 14, 2019 at 6:00 PM Chesnay Schepler < >> > >> ches...@apache.org >> > >>>>>>>> <mailto:ches...@apache.org>> wrote: >> > >>>>>>>> >> > >>>>>>>> So here's the state of things: >> > >>>>>>>> >> > >>>>>>>> >> > >>>>>>>> The master of flink-shaded now uses jackson 2.10.1, >> which >> > >>>>>>>> eliminates a whole category of security vulnerabilities. >> > >>>>>>>> The flink master works perfectly fine with that version; >> > 1.9 >> > >> will >> > >>>>>>>> likely do so too and 1.8 would require a minor >> adjustment. >> > >>>>>>>> >> > >>>>>>>> Hence, there may be value in first doing a flink-shaded >> > >>>>>>>> release so >> > >>>>>>>> we can eliminate these vulnerabilities in 1.8.3 and >> 1.9.2 . >> > >>>>>>>> >> > >>>>>>>> >> > >>>>>>>> As for other jackson dependencies (coming from calcite, >> > kafka, >> > >>>>>>>> kinesis), I ran the unit and end-to-end tests of master >> > >> yesterday >> > >>>>>>>> will /all /jackson dependencies set to 2.10.1, and they >> > >> passed. I >> > >>>>>>>> will open a PR soon-ish for making this change on >> master. >> > >>>>>>>> >> > >>>>>>>> The question now is whether we want to backport this >> > change to >> > >>>>>>>> 1.8/1.9 . >> > >>>>>>>> Some code paths /may /not be covered by our tests, and >> > >> transitive >> > >>>>>>>> jackson users /might /run into issues. >> > >>>>>>>> Alternatively, we could set this up as an opt-in >> upgrade, >> > by >> > >>>>>>>> adding a separate profile that bumps the versions. This >> > would >> > >>>>>>>> present users/providers who are concerned about the >> > >>>>>>>> vulnerabilities an easy workaround, at the risk of /some >> > >> /things >> > >>>>>>>> /maybe /not working. >> > >>>>>>>> >> > >>>>>>>> On 14/11/2019 03:16, Hequn Cheng wrote: >> > >>>>>>>>> Hi Chesnay, Jincheng >> > >>>>>>>>> >> > >>>>>>>>> Sure, I think it's good to have these fixes. >> > >>>>>>>>> Thanks a lot for providing the information about the >> > security >> > >>>>>>>>> vulnerabilities! @Chesnay >> > >>>>>>>>> >> > >>>>>>>>> Best, Hequn >> > >>>>>>>>> >> > >>>>>>>>> On Thu, Nov 14, 2019 at 10:07 AM jincheng sun< >> > >>>>>> sunjincheng...@gmail.com> <mailto:sunjincheng...@gmail.com> >> > >>>>>>>>> wrote: >> > >>>>>>>>> >> > >>>>>>>>>> +1 for try to eliminate the security vulnerabilities. >> > Great >> > >>>>>> thanks for >> > >>>>>>>>>> doing this important work, Chesnay! >> > >>>>>>>>>> What do you think Hequn ? >> > >>>>>>>>>> >> > >>>>>>>>>> Best, >> > >>>>>>>>>> Jincheng >> > >>>>>>>>>> >> > >>>>>>>>>> Chesnay Schepler<ches...@apache.org> >> > >>>>>>>>>> <mailto:ches...@apache.org> >> > >>>>>> 于2019年11月13日周三 下午5:17写道： >> > >>>>>>>>>>> It would be great if you could give me a day or 2 to >> > check >> > >> how >> > >>>>>> easy it >> > >>>>>>>>>>> would be to bump the various jackson dependencies to >> > >>>>>>>>>>> eliminate a >> > >>>>>> few >> > >>>>>>>>>>> security vulnerabilities. >> > >>>>>>>>>>> >> > >>>>>>>>>>> On 09/11/2019 05:10, jincheng sun wrote: >> > >>>>>>>>>>>> Hi Flink devs, >> > >>>>>>>>>>>> >> > >>>>>>>>>>>> It has been more than 2 months since the 1.8.2 >> > released. >> > >> So, >> > >>>>>> What do >> > >>>>>>>>>> you >> > >>>>>>>>>>>> think about releasing Flink 1.8.3 soon? >> > >>>>>>>>>>>> >> > >>>>>>>>>>>> We already have many important bug fixes in the >> > >> release-1.8 >> > >>>>>> branch (29 >> > >>>>>>>>>>>> resolved issues). >> > >>>>>>>>>>>> >> > >>>>>>>>>>>> Most notable fixes are: >> > >>>>>>>>>>>> >> > >>>>>>>>>>>> - FLINK-14010 Dispatcher & JobManagers don't give up >> > >>>>>>>>>>>> leadership >> > >>>>>> when AM >> > >>>>>>>>>>> is >> > >>>>>>>>>>>> shut down >> > >>>>>>>>>>>> - FLINK-14315 NPE with >> JobMaster.disconnectTaskManager >> > >>>>>>>>>>>> - FLINK-12848 Method equals() in RowTypeInfo should >> > >> consider >> > >>>>>>>>>> fieldsNames >> > >>>>>>>>>>>> - FLINK-12342 Yarn Resource Manager Acquires Too >> Many >> > >>>>>>>>>>>> Containers >> > >>>>>>>>>>>> - FLINK-14589 Redundant slot requests with the same >> > >>>>>> AllocationID leads >> > >>>>>>>>>> to >> > >>>>>>>>>>>> inconsistent slot table >> > >>>>>>>>>>>> >> > >>>>>>>>>>>> Furthermore, the following critical issues is in >> > progress, >> > >>>>>> maybe we can >> > >>>>>>>>>>>> wait for it if it is not too much effort. >> > >>>>>>>>>>>> >> > >>>>>>>>>>>> - FLINK-13184 Starting a TaskExecutor blocks the >> > >>>>>> YarnResourceManager's >> > >>>>>>>>>>> main >> > >>>>>>>>>>>> thread >> > >>>>>>>>>>>> >> > >>>>>>>>>>>> Please let me know what you think? >> > >>>>>>>>>>>> >> > >>>>>>>>>>>> Best, >> > >>>>>>>>>>>> Jincheng >> > >>>>>>>>>>>> >> > >> >> > >> > >> >
