Sorry for the late reply. +1 for the general proposal.
And one remainder, to use UNKNOWN resource requirement, we need to make sure optimizer knowns which operators use off-heap managed memory, and compute and set a fraction to the operators. See FLIP-53[1] for more details, and I would suggest you to double check with @Zhu Zhu who works on this part. Thank you~ Xintong Song [1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-53%3A+Fine+Grained+Operator+Resource+Management On Tue, Nov 12, 2019 at 11:53 AM Dian Fu <dian0511...@gmail.com> wrote: > Hi Jincheng, > > Thanks for the reply and also looking forward to the feedback from the > community. > > Thanks, > Dian > > > 在 2019年11月11日，下午2:34，jincheng sun <sunjincheng...@gmail.com> 写道： > > > > Hi all, > > > > +1， Thanks for bring up this discussion Dian! > > > > The Resource Management is very important for PyFlink UDF. So, It's great > > if anyone can add more comments or inputs in the design doc or feedback > in > > ML. :) > > > > Best, > > Jincheng > > > > Dian Fu <dian0511...@gmail.com> 于2019年11月5日周二 上午11:32写道： > > > >> Hi everyone, > >> > >> In FLIP-58[1] it will add the support of Python user-defined stateless > >> function for Python Table API. It will launch a separate Python process > for > >> Python user-defined function execution. The resources used by the Python > >> process should be managed properly by Flink’s resource management > >> framework. FLIP-49[2] has proposed a unified memory management framework > >> and PyFlink user-defined function resource management should be based on > >> it. Jincheng, Hequn, Xintong, GuoWei and I discussed offline about > this. I > >> draft a design doc[3] and want to start a discussion about PyFlink > >> user-defined function resource management. > >> > >> Welcome any comments on the design doc or giving us feedback on the ML > >> directly. > >> > >> Regards, > >> Dian > >> > >> [1] > >> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-58%3A+Flink+Python+User-Defined+Stateless+Function+for+Table > >> [2] > >> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-49%3A+Unified+Memory+Configuration+for+TaskExecutors > >> [3] > >> > https://docs.google.com/document/d/1LQP8L66Thu2yVv6YRSfmF9EkkMnwhBHGjcTQ11GUmFc/edit#heading=h.4q4ggaftf78m > >
