There is still no such optimization at framework level. However, I think this maybe a good point that we could optimize. Would you like to create a ticket for this?
Regards, Dian > 2021年5月8日 下午2:27,Yik San Chan <evan.chanyik...@gmail.com> 写道: > > Hi Dian, > > Thanks for pointing that out, it is a work around that I have also considered. > > I wonder if there is a framework level optimization on this, so that a UDF is > only initiated once, no matter how many times it is called? > > Thank you! > > Best, > Yik San > > On Sat, May 8, 2021 at 1:32 PM Dian Fu <dian0511...@gmail.com > <mailto:dian0511...@gmail.com>> wrote: > Hi Yik San, > > Is it acceptable to rewrite the UDF a bit to accept multiple parameters and > then rewrite the program as following: > > ``` > SELECT > LABEL_ENCODE(a, b, c) > ... > ``` > > Regards, > Dian > >> 2021年5月8日 上午11:56,Yik San Chan <evan.chanyik...@gmail.com >> <mailto:evan.chanyik...@gmail.com>> 写道: >> >> Hi community, >> >> I am using PyFlink and Pandas UDF in my job. >> >> The job executes a SQL like this: >> >> ``` >> SELECT >> LABEL_ENCODE(a), >> LABEL_ENCODE(b), >> LABEL_ENCODE(c) >> ... >> ``` >> >> And my LABEL_ENCODE UDF is defined below: >> >> ``` >> class LabelEncode(ScalarFunction): >> def open(self, function_context): >> logging.info <http://logging.info/>("LabelEncode.open") >> self.encoder = load_encoder() >> def eval(self, x): >> ... >> >> labelEncode = udf(LabelEncode(), ...) >> ``` >> >> When I run the job, according to taskmanger log, "LabelEncode.open" is >> printed 3 times, which is exactly the times LABEL_ENCODE udf is called. >> >> Since every LabelEncode.open causes an I/O (load_encoder() does so), I >> wonder if I can only initiate the UDF once, and use it 3 times? >> >> Thank you! >> >> Best, >> Yik San >