Hi, @Ufuk Celebi <u...@apache.org> Hi, we are very close now. There is one issue( FLINK-13995 <https://issues.apache.org/jira/browse/FLINK-13995>) left that I want to double-check with you guys. Once this is done, we can create the first RC. I already have some minor comments in the PR <https://github.com/apache/flink/pull/10195>.
@Zhu Zhu <reed...@gmail.com> Glad to hear that it is not a blocker. Thank you. Best, Hequn On Mon, Nov 25, 2019 at 5:43 PM Ufuk Celebi <u...@apache.org> wrote: > @Hequn: flink-shaded:9.0 is available in Maven central now. I think you > can go ahead and create the first RC. :-) > > On Mon, Nov 25, 2019 at 7:47 AM Zhu Zhu <reed...@gmail.com> wrote: > >> Hi Hequn, >> >> Looks we are not able to merge fix of FLINK-14735 to 1.8 very soon. >> Given that this fix is for batch job only and batch is not very good in >> 1.8, I think it is a not blocker of release 1.8.3. >> So just don't be blocked by it and feel free to cut the RC when other >> blocking issues are resolved. >> >> Thanks, >> Zhu Zhu >> >> Hequn Cheng <chenghe...@gmail.com> 于2019年11月23日周六 下午9:08写道： >> >> > Hi Zhu Zhu, >> > >> > Thanks a lot for letting us know! >> > We can't cut the first RC right now due to the wait of the flink-shade >> > release, so go ahead. >> > >> > Theoretically, we will cut the first RC of 1.8.3 and vote for it once >> the >> > release of flink-shade is done, >> > but I will try my best to have it in 1.8.3. Hope we can get it on board >> on >> > time. :) >> > >> > Best, Hequn >> > >> > On Sat, Nov 23, 2019 at 10:40 AM Zhu Zhu <reed...@gmail.com> wrote: >> > >> >> Hi Jincheng & Hequn >> >> >> >> Thanks for driving the releasing of 1.8.3. >> >> >> >> I am now working on FLINK-14735. The fix avoids duplicated input >> >> checking when scheduling ALL-to-ALL >> >> connected downstream consumers with ALL input constraints. The >> duplicated >> >> checking can cause severe >> >> performance issues for large scale jobs. So I hope the fix could be >> >> released with 1.8.3. >> >> >> >> The fix is already merged into master, and is now in the process of >> >> backporting to 1.8. >> >> >> >> Thanks, >> >> Zhu Zhu >> >> >> >> Ufuk Celebi <u...@apache.org> 于2019年11月15日周五 下午11:54写道： >> >> >> >>> Thanks Chesnay. >> >>> >> >>> I'm also +1 to release 1.8.3 asap without the changes for the Jackson >> >>> version bump and leave those for a future release. Realistically, the >> >>> flink-shaded release will take until mid next week or end of next >> week. >> >>> But >> >>> please correct me if you think that it should not take that long or >> it's >> >>> OK >> >>> to block the 1.8.3 release on the flink-shaded release. >> >>> >> >>> – Ufuk >> >>> >> >>> >> >>> On Fri, Nov 15, 2019 at 2:27 PM Chesnay Schepler <ches...@apache.org> >> >>> wrote: >> >>> >> >>> > I've kicked off a discussion about the next flink-shaded release, >> and >> >>> > have opened PRs for adding the opt-in profile to 1.8/1.9. >> >>> > >> >>> > On 15/11/2019 13:54, Hequn Cheng wrote: >> >>> > > That's great, thank you very much! Ideally, we can kick off the >> >>> release >> >>> > > vote for the first RC of 1.8.3 within next week. :) >> >>> > > >> >>> > > On Fri, Nov 15, 2019 at 8:47 PM Chesnay Schepler < >> ches...@apache.org >> >>> > >> >>> > wrote: >> >>> > > >> >>> > >> I'm not aware of any more planned changes to flink-shaded; so we >> >>> could >> >>> > >> start the release right away. >> >>> > >> >> >>> > >> On 15/11/2019 13:44, Hequn Cheng wrote: >> >>> > >>> Hi, >> >>> > >>> >> >>> > >>> @Chesnay Thanks a lot for the explanation. +1 to the opt-in >> >>> approach >> >>> > for >> >>> > >>> 1.8/1.9. >> >>> > >>> @Ufuk Thank you for the nice summary. >> >>> > >>> >> >>> > >>> Looks good so far except that we need to postpone 1.8.3 a bit to >> >>> first >> >>> > >> do a >> >>> > >>> flink-shaded release. >> >>> > >>> BTW, @chesnay when would we plan to release the flink-shaded >> with >> >>> > >> upgraded >> >>> > >>> Jackson? >> >>> > >>> >> >>> > >>> Best, Hequn >> >>> > >>> >> >>> > >>> On Fri, Nov 15, 2019 at 7:43 PM Chesnay Schepler < >> >>> ches...@apache.org> >> >>> > >> wrote: >> >>> > >>>> One small modification: the flink-shaded upgrade does not have >> to >> >>> be >> >>> > >>>> part of the profile; since it is only intended for internal use >> >>> anyway >> >>> > >>>> (and thus has limited exposure) we can be pretty sure this >> doesn't >> >>> > break >> >>> > >>>> anything. >> >>> > >>>> >> >>> > >>>> On 15/11/2019 12:23, Chesnay Schepler wrote: >> >>> > >>>>> Ufuk's summary is correct. >> >>> > >>>>> >> >>> > >>>>> There's a slight caveat in that we'd also have to bump the >> >>> > >>>>> shade-plugin to 3.1.1 since it otherwise fails on jackson, >> >>> > >>>>> but I have no concerns about this change. >> >>> > >>>>> >> >>> > >>>>> On 15/11/2019 12:19, Ufuk Celebi wrote: >> >>> > >>>>>> The opt-in approach seems reasonable to me. +1 to include the >> >>> > >>>>>> profiles in >> >>> > >>>>>> 1.8 and 1.9 without changing the default versions (including >> the >> >>> > >> default >> >>> > >>>>>> version of flink-shaded). >> >>> > >>>>>> >> >>> > >>>>>> As far as I can tell, the next steps would be: >> >>> > >>>>>> >> >>> > >>>>>> 1) Release flink-shaded with upgraded Jackson >> >>> > >>>>>> 2a) Bump the flink-shaded version by default in master >> >>> > >>>>>> 2b) Create opt-in profiles for 1.8 and 1.9 (the opt-in >> profiles >> >>> > >>>>>> should also >> >>> > >>>>>> cover the upgrade to the most recent flink-shaded version) >> >>> > >>>>>> >> >>> > >>>>>> @Chesnay: is this a correct summary? >> >>> > >>>>>> >> >>> > >>>>>> Note this would block the 1.8.3 release on step 1. As an >> >>> upside, we >> >>> > >>>>>> might >> >>> > >>>>>> get some additional feedback until the 1.10 release with >> these >> >>> > >>>>>> profiles in >> >>> > >>>>>> case users make use of them with 1.8/1.9. >> >>> > >>>>>> >> >>> > >>>>>> – Ufuk >> >>> > >>>>>> >> >>> > >>>>>> On Fri, Nov 15, 2019 at 12:08 PM Chesnay Schepler < >> >>> > ches...@apache.org >> >>> > >>>>>> wrote: >> >>> > >>>>>>> The opt-in approach would only be used for 1.8.3 / 1.9.2; on >> >>> master >> >>> > >>>>>>> (and >> >>> > >>>>>>> thus starting from 1.10.0) it's not opt-in. >> >>> > >>>>>>> >> >>> > >>>>>>> I have only proposed it as an opt-in because a) we usually >> do >> >>> not >> >>> > >> bump >> >>> > >>>>>>> dependencies in bugfix releases and b) it's a short-term >> change >> >>> > that >> >>> > >> we >> >>> > >>>>>>> aren't allowing to mature properly. >> >>> > >>>>>>> In contrast, the 1.10 release is significantly further away, >> >>> hence >> >>> > no >> >>> > >>>>>>> opt-in. >> >>> > >>>>>>> >> >>> > >>>>>>> Hence, I'm not concerned about such kind of ugprades being >> more >> >>> > >> common >> >>> > >>>>>>> in the future. >> >>> > >>>>>>> >> >>> > >>>>>>> We can certainly support every jackson version that fixes >> these >> >>> > >>>>>>> vulnerabilities; individual modules can always use a >> different >> >>> > >> version >> >>> > >>>>>>> (that hopefully includes the fixes). >> >>> > >>>>>>> Ideally of course we'd only be using 1 version, but that may >> >>> or may >> >>> > >> not >> >>> > >>>>>>> be feasible. >> >>> > >>>>>>> >> >>> > >>>>>>> On 15/11/2019 04:07, Hequn Cheng wrote: >> >>> > >>>>>>>> Hi Chesnay, >> >>> > >>>>>>>> >> >>> > >>>>>>>> Great to hear that jackson-2.10.1 works well on master. >> >>> Really a >> >>> > >> good >> >>> > >>>>>> job! >> >>> > >>>>>>>> - Whether backport this change to 1.8/1.9 >> >>> > >>>>>>>> I had taken a quick look at the security vulnerabilities, >> >>> some of >> >>> > >> them >> >>> > >>>>>>>> seem can lead to high-security problems, thus from my >> point of >> >>> > view, >> >>> > >>>>>>>> I'm in favor of adding the fix into 1.9/1.8. However, I >> would >> >>> like >> >>> > >> to >> >>> > >>>>>>>> trust your judgment as you are more professional at this >> >>> problem. >> >>> > >>>>>>>> >> >>> > >>>>>>>> - How to port this change to 1.8/1.9 >> >>> > >>>>>>>> I think providing an opt-in upgrade is a good idea. Another >> >>> > question >> >>> > >>>>>>>> here is whether do we plan to support multi jackson >> versions >> >>> that >> >>> > >> have >> >>> > >>>>>>>> eliminated the security vulnerabilities. If we only plan to >> >>> > support >> >>> > >>>>>>>> 2.10.1, I would like to make it a non-opt-in upgrade. As an >> >>> > option, >> >>> > >>>>>>>> users can downgrade the flink version if meet problems >> using >> >>> the >> >>> > new >> >>> > >>>>>>>> version. Of course, we will try our best to make the new >> >>> release >> >>> > out >> >>> > >>>>>>>> of question. >> >>> > >>>>>>>> Another concern of making it an opt-in upgrade is, it will >> >>> make >> >>> > our >> >>> > >>>>>>>> build unlikely convergence as more and more build options >> >>> will be >> >>> > >>>>>>>> added when we upgrade a commonly used lib like this one. >> >>> > >>>>>>>> >> >>> > >>>>>>>> What do you think? >> >>> > >>>>>>>> >> >>> > >>>>>>>> Best, Hequn >> >>> > >>>>>>>> >> >>> > >>>>>>>> On Thu, Nov 14, 2019 at 6:00 PM Chesnay Schepler < >> >>> > >> ches...@apache.org >> >>> > >>>>>>>> <mailto:ches...@apache.org>> wrote: >> >>> > >>>>>>>> >> >>> > >>>>>>>> So here's the state of things: >> >>> > >>>>>>>> >> >>> > >>>>>>>> >> >>> > >>>>>>>> The master of flink-shaded now uses jackson 2.10.1, >> >>> which >> >>> > >>>>>>>> eliminates a whole category of security >> >>> vulnerabilities. >> >>> > >>>>>>>> The flink master works perfectly fine with that >> >>> version; >> >>> > 1.9 >> >>> > >> will >> >>> > >>>>>>>> likely do so too and 1.8 would require a minor >> >>> adjustment. >> >>> > >>>>>>>> >> >>> > >>>>>>>> Hence, there may be value in first doing a >> flink-shaded >> >>> > >>>>>>>> release so >> >>> > >>>>>>>> we can eliminate these vulnerabilities in 1.8.3 and >> >>> 1.9.2 . >> >>> > >>>>>>>> >> >>> > >>>>>>>> >> >>> > >>>>>>>> As for other jackson dependencies (coming from >> calcite, >> >>> > kafka, >> >>> > >>>>>>>> kinesis), I ran the unit and end-to-end tests of >> master >> >>> > >> yesterday >> >>> > >>>>>>>> will /all /jackson dependencies set to 2.10.1, and >> they >> >>> > >> passed. I >> >>> > >>>>>>>> will open a PR soon-ish for making this change on >> >>> master. >> >>> > >>>>>>>> >> >>> > >>>>>>>> The question now is whether we want to backport this >> >>> > change to >> >>> > >>>>>>>> 1.8/1.9 . >> >>> > >>>>>>>> Some code paths /may /not be covered by our tests, >> and >> >>> > >> transitive >> >>> > >>>>>>>> jackson users /might /run into issues. >> >>> > >>>>>>>> Alternatively, we could set this up as an opt-in >> >>> upgrade, >> >>> > by >> >>> > >>>>>>>> adding a separate profile that bumps the versions. >> This >> >>> > would >> >>> > >>>>>>>> present users/providers who are concerned about the >> >>> > >>>>>>>> vulnerabilities an easy workaround, at the risk of >> >>> /some >> >>> > >> /things >> >>> > >>>>>>>> /maybe /not working. >> >>> > >>>>>>>> >> >>> > >>>>>>>> On 14/11/2019 03:16, Hequn Cheng wrote: >> >>> > >>>>>>>>> Hi Chesnay, Jincheng >> >>> > >>>>>>>>> >> >>> > >>>>>>>>> Sure, I think it's good to have these fixes. >> >>> > >>>>>>>>> Thanks a lot for providing the information about >> the >> >>> > security >> >>> > >>>>>>>>> vulnerabilities! @Chesnay >> >>> > >>>>>>>>> >> >>> > >>>>>>>>> Best, Hequn >> >>> > >>>>>>>>> >> >>> > >>>>>>>>> On Thu, Nov 14, 2019 at 10:07 AM jincheng sun< >> >>> > >>>>>> sunjincheng...@gmail.com> <mailto:sunjincheng...@gmail.com> >> >>> > >>>>>>>>> wrote: >> >>> > >>>>>>>>> >> >>> > >>>>>>>>>> +1 for try to eliminate the security >> vulnerabilities. >> >>> > Great >> >>> > >>>>>> thanks for >> >>> > >>>>>>>>>> doing this important work, Chesnay! >> >>> > >>>>>>>>>> What do you think Hequn ? >> >>> > >>>>>>>>>> >> >>> > >>>>>>>>>> Best, >> >>> > >>>>>>>>>> Jincheng >> >>> > >>>>>>>>>> >> >>> > >>>>>>>>>> Chesnay Schepler<ches...@apache.org> >> >>> > >>>>>>>>>> <mailto:ches...@apache.org> >> >>> > >>>>>> 于2019年11月13日周三 下午5:17写道： >> >>> > >>>>>>>>>>> It would be great if you could give me a day or >> 2 to >> >>> > check >> >>> > >> how >> >>> > >>>>>> easy it >> >>> > >>>>>>>>>>> would be to bump the various jackson >> dependencies to >> >>> > >>>>>>>>>>> eliminate a >> >>> > >>>>>> few >> >>> > >>>>>>>>>>> security vulnerabilities. >> >>> > >>>>>>>>>>> >> >>> > >>>>>>>>>>> On 09/11/2019 05:10, jincheng sun wrote: >> >>> > >>>>>>>>>>>> Hi Flink devs, >> >>> > >>>>>>>>>>>> >> >>> > >>>>>>>>>>>> It has been more than 2 months since the 1.8.2 >> >>> > released. >> >>> > >> So, >> >>> > >>>>>> What do >> >>> > >>>>>>>>>> you >> >>> > >>>>>>>>>>>> think about releasing Flink 1.8.3 soon? >> >>> > >>>>>>>>>>>> >> >>> > >>>>>>>>>>>> We already have many important bug fixes in the >> >>> > >> release-1.8 >> >>> > >>>>>> branch (29 >> >>> > >>>>>>>>>>>> resolved issues). >> >>> > >>>>>>>>>>>> >> >>> > >>>>>>>>>>>> Most notable fixes are: >> >>> > >>>>>>>>>>>> >> >>> > >>>>>>>>>>>> - FLINK-14010 Dispatcher & JobManagers don't >> give >> >>> up >> >>> > >>>>>>>>>>>> leadership >> >>> > >>>>>> when AM >> >>> > >>>>>>>>>>> is >> >>> > >>>>>>>>>>>> shut down >> >>> > >>>>>>>>>>>> - FLINK-14315 NPE with >> >>> JobMaster.disconnectTaskManager >> >>> > >>>>>>>>>>>> - FLINK-12848 Method equals() in RowTypeInfo >> should >> >>> > >> consider >> >>> > >>>>>>>>>> fieldsNames >> >>> > >>>>>>>>>>>> - FLINK-12342 Yarn Resource Manager Acquires Too >> >>> Many >> >>> > >>>>>>>>>>>> Containers >> >>> > >>>>>>>>>>>> - FLINK-14589 Redundant slot requests with the >> same >> >>> > >>>>>> AllocationID leads >> >>> > >>>>>>>>>> to >> >>> > >>>>>>>>>>>> inconsistent slot table >> >>> > >>>>>>>>>>>> >> >>> > >>>>>>>>>>>> Furthermore, the following critical issues is in >> >>> > progress, >> >>> > >>>>>> maybe we can >> >>> > >>>>>>>>>>>> wait for it if it is not too much effort. >> >>> > >>>>>>>>>>>> >> >>> > >>>>>>>>>>>> - FLINK-13184 Starting a TaskExecutor blocks the >> >>> > >>>>>> YarnResourceManager's >> >>> > >>>>>>>>>>> main >> >>> > >>>>>>>>>>>> thread >> >>> > >>>>>>>>>>>> >> >>> > >>>>>>>>>>>> Please let me know what you think? >> >>> > >>>>>>>>>>>> >> >>> > >>>>>>>>>>>> Best, >> >>> > >>>>>>>>>>>> Jincheng >> >>> > >>>>>>>>>>>> >> >>> > >> >> >>> > >> >>> > >> >>> >> >> >> >
