Hi, you may first enable the Kryo fallback option first, submit the job, and search for "be processed as GenericType". Flink will print it in most cases where we fall back to Kryo (a few exceptions including type Class, Object, recursive types, interface).
Best, Zhanghao Chen ________________________________ From: Salva Alcántara <salcantara...@gmail.com> Sent: Monday, April 8, 2024 16:01 To: Yunfeng Zhou <flink.zhouyunf...@gmail.com> Cc: user@flink.apache.org <user@flink.apache.org> Subject: Re: Debugging Kryo Fallback Yeah I think you're right and there is no need for anything, really. I was thinking of having more user friendly tests for my POJOs for which I checked the Kryo Fallback and if detected provide an exhaustive list of issues found (vs raising an exception for the first problem, requiring users to fix it, then see if there are more issues/exceptions, fix them and so on). Salva On Mon, Apr 8, 2024 at 8:46 AM Yunfeng Zhou <flink.zhouyunf...@gmail.com<mailto:flink.zhouyunf...@gmail.com>> wrote: Hi Salva, Could you please give me some hint about the issues Flink can collect apart from the exception and the existing logs? Suppose we record the exception in the log and the Flink job continues, I can imagine that similar Kryo exceptions from each of the rest records will then appear in the log as well. They expose the same bug to fix so there is no need to collect all of them. Would there be other issues to collect apart from these repeating similar exceptions? Best regards, Yunfeng On Sun, Apr 7, 2024 at 5:15 PM Salva Alcántara <salcantara...@gmail.com<mailto:salcantara...@gmail.com>> wrote: > > Thanks Yunfeng! That is more or less what I do now when I run into the > problem. This approach reports problems one at a time (an exception is raised > on the first problem encountered). > > Instead of that, I think accumulating all the issues and presenting them all > at once would be more user friendly. > > Regards, > > Salva > > On Sun, Apr 7, 2024 at 5:43 AM Yunfeng Zhou > <flink.zhouyunf...@gmail.com<mailto:flink.zhouyunf...@gmail.com>> wrote: >> >> Hi Salva, >> >> According to the description of the configuration >> `pipeline.generic-types`, after setting this to false you should be >> able to find UnsupportedOperationException in the Flink log describing >> the data types that have not been supported in Kryo. Then you should >> be able to look into your code finding out the usages of the certain >> data type and perform corresponding bug fixes. Other information >> provided in Flink's log, like those info-level logs in >> TypeExtractor@analyzePojo, might help reveal more details around the >> exception. >> >> Best, >> Yunfeng >> >> On Wed, Apr 3, 2024 at 4:19 PM Salva Alcántara >> <salcantara...@gmail.com<mailto:salcantara...@gmail.com>> wrote: >> > >> > FYI Reposted in SO: >> > - >> > https://stackoverflow.com/questions/78265380/how-to-debug-the-kryo-fallback-in-flink >> > >> > On Thu, Mar 28, 2024 at 7:24 AM Salva Alcántara >> > <salcantara...@gmail.com<mailto:salcantara...@gmail.com>> wrote: >> >> >> >> I wonder which is the simplest way of troubleshooting/debugging what >> >> causes the Kryo fallback. >> >> >> >> Detecting it is just a matter of adding this line to your job: >> >> >> >> ``` >> >> env.getConfig().disableGenericTypes(); >> >> ``` >> >> >> >> or in more recent versions: >> >> >> >> ``` >> >> pipeline.generic-types: false >> >> >> >> ``` >> >> >> >> But once you detect the issue, what is the simplest way to debug it? You >> >> can of course add a breakpoint in: >> >> org.apache.flink.api.java.typeutils.TypeExtractor@analyzePojo >> >> >> >> but ideally there should be a simpler way to show all the problems >> >> encountered to the user without having to get that deep into the code. >> >> >> >> Thanks in advance, >> >> >> >> Salva
