Hi Dawid, Thanks again for replying and for confirming the behavior.
Best regards, Joel On Tue, Sep 7, 2021 at 1:35 PM Dawid Wysakowicz <dwysakow...@apache.org> wrote: > Huh, of course. Actually I was too quick with my answer. Even if it is > serialized with the JobGraph, the class is necessary on TMs to be > deserialized. That's how java serialization works after all. > > So the actual answer, it is serialized with the JobGraph. The class is > mandatory for deserialization. > > Best, > > Dawid > On 07/09/2021 16:31, Joel Edwards wrote: > > Good day, > > I have been attempting to submit a job to a session cluster. This job > involves a pair of dynamic tables and a SQL query. The SQL query is calling > a UDF which I register via the table API's createTemporarySystemFunction() > method. The job runs locally, but when I attempt to submit it to a remote > session cluster, the job fails with the error: > > `Cannot load user class: <fully-qualified-class-name>` > > If I place a fat jar containing all of my local dependencies on the > JobManagers and TaskManagers, the UDF will be loaded. However, I would > expect the UDF to be serialized and sent with the rest of the job. I have > looked over the UDF documentation, and I don't see a reason why it would > not be serialized with the rest of the job. However, seeing as there is no > error related to serializing the UDF, my assumptions related to UDF > serialization must be incorrect. Is there a hint I can use to cause the > closure cleaner to identify the UDF for serialization? I suspect the reason > it is not being included is that it is referenced only in the SQL query, > and not streams feeding the input table or the stream consuming the output > table. > > Summary of questions: > - Will UDF be serialized with the job? Or are they never included? > - Is it possible to hint at what should be serialized and sent along with > the job? > > Thank you, > Joel > > > -- > Joel Edwards > Software Architect > Ed-Craft Software Solutions > > -- Joel Edwards Software Architect Ed-Craft Software Solutions
