Hi, According to this blog post https://flink.apache.org/2020/04/15/flink-serialization-tuning-vol.-1-choosing-your-serializer-if-you-can/#pojoserializer <https://flink.apache.org/2020/04/15/flink-serialization-tuning-vol.-1-choosing-your-serializer-if-you-can/#pojoserializer> The "Must be processed as GenericType" message means that the POJO serializer will not be used and instead, Kyro will be used.
I created a simple POJO to test it again with a java Collection but I got the same message. Disabling generic types throws an exception. I'm not sure how to use these types along with the POJO serializer or any other fast serializer. Best regards, Saleh. > On 14 Aug 2023, at 4:59 AM, liu ron <ron9....@gmail.com> wrote: > > Hi, > > According to the test in [1], I think Flink can recognize Pojo class which > contains java List, so I think you can refer to the related Pojo class > implementation. > > [1] > https://github.com/apache/flink/blob/fc2b5d8f53a41695117f6eaf4c798cc183cf1e36/flink-core/src/test/java/org/apache/flink/api/java/typeutils/PojoTypeExtractionTest.java#L192 > > <https://github.com/apache/flink/blob/fc2b5d8f53a41695117f6eaf4c798cc183cf1e36/flink-core/src/test/java/org/apache/flink/api/java/typeutils/PojoTypeExtractionTest.java#L192> > > Best, > Ron > > <s...@sammar.sa <mailto:s...@sammar.sa>> 于2023年8月13日周日 22:50写道： > Greetings, > > I am working on a project that needs to process around 100k events per second > and I'm trying to improve performance. > > Most of the classes being used are POJOs but have a couple of fields using a > `java.util` class, either `ArrayList`, `HashSet` or `SortedSet` etc. This > forces Flink to use Kyro and throw these warnings: > > ``` > class java.util.ArrayList does not contain a setter for field size > Class class java.util.ArrayList cannot be used as a POJO type because not all > fields are valid POJO fields, and must be processed as GenericType. Please > read the Flink documentation on "Data Types & Serialization" for details of > the effect on performance and schema evolution. > ``` > > ``` > No fields were detected for class java.util.HashSet so it cannot be used as a > POJO type and must be processed as GenericType. Please read the Flink > documentation on "Data Types & Serialization" for details of the effect on > performance and schema evolution. > I read through the documentation and stackoverflow and the conclusion is that > I need to make a TypeInfoFactory and use it inside a TypeInfo annotation over > my POJO. > ``` > > My question is what do I need to do to get Flink to recognize my classes as > POJOs and use the POJO serializer for better performance? > I read through the documentation and stackoverflow and the conclusion is that > I need to make a TypeInfoFactory and use it inside a TypeInfo annotation over > my POJO. > While this seems incredibly tedious and I keep thinking "there must be a > better way", I would be fine with this solution if I could figure out how to > do this for the Set types I'm using. > > Any help would be appreciated.
