Hi all,
Thanks again for the help! I have one more follow up question regarding
Flink and serialization on v1.18. I know state schema evolution is
supported for POJOs in Flink. However, if my class uses the POJO serializer
but has a field that falls back to Kryo (such as UUID), would it still be
eligible for Flink’s state schema evolution support?
Best,
Richard
On Fri, May 16, 2025 at 3:59 AM Zhanghao Chen <zhanghao.c...@outlook.com>
wrote:
> Flink 2.0 will work. You may use Types.LIST for lists and Types.MAP for
> sets (mocked by a Map) for that. Notice that Flink's built-in LIST does
> not support null element and MAP type does not support null key, and
> neither support null collection. In Flink 2.0, we've added special
> treatment of these nullable cases.
>
> Best,
> Zhanghao Chen
> ------------------------------
> *From:* Мосин Николай <mosin...@yandex.ru>
> *Sent:* Friday, May 16, 2025 0:02
> *To:* Richard Cheung <rcheungsi...@gmail.com>
> *Cc:* Schwalbe Matthias <matthias.schwa...@viseca.ch>; Zhanghao Chen <
> zhanghao.c...@outlook.com>; user@flink.apache.org <user@flink.apache.org>
> *Subject:* Re: Apache Flink Serialization Question
>
> For List I just setup TypeInfoFactory like:
>
> public class SomeDataDTOTypeInfoFactory extends
> TypeInfoFactory<SomeDataDTO> {
> @Override
> public TypeInformation<SomeDataDTO> createTypeInfo(Type t, Map<String,
> TypeInformation<?>> genericParameters) {
> return Types.POJO(SomeDataDTO.class, new HashMap<String,
> TypeInformation<?>>() {
> {
> ....
> put("sessionId", UUIDTypeInfo.INSTANCE);
> put("tags", Types.LIST(Types.STRING));
> ....
> }
> });
> }
> }
>
> and annotate DTO
> @TypeInfo(SomeDataDTOTypeInfoFactory.class)
> public class SomeDataDTO {
> ....
> private UUID sessionId;
> private List<String> tags = new ArrayList<>();
> ....
>
> But for Set I don't found workaround and as I understand it must be
> replaced by List
>
>
>
>
> ----------------
> Кому: Mosin Nick (mosin...@yandex.ru);
> Копия: Schwalbe Matthias (matthias.schwa...@viseca.ch), Zhanghao Chen (
> zhanghao.c...@outlook.com), user@flink.apache.org;
> Тема: Apache Flink Serialization Question;
> 15.05.2025, 18:27, "Richard Cheung" <rcheungsi...@gmail.com>:
>
> Thanks for all the replies!
>
> I’ve decided to just update my UUID field to a String for POJO compliance.
> However, I’m getting the same log issues for List<String> and Set<String>
> saying that fields will be processed as GenericType. I want everything to
> be fully POJO compatible so I can have schema evolution for the future. Is
> there a workaround for this for POJO compliance in Flink v1.8 or would I
> have to upgrade to Flink v2 which supports common collection types for
> serialization or maybe the even upgrading to v2 won’t work?
>
> Best regards,
> Richard
>
> On Thu, May 15, 2025 at 9:06 AM Mosin Nick <mosin...@yandex.ru> wrote:
>
> Github already contain some serializer for UUID
> https://github.com/gAmUssA/datalorean/blob/main/src/main/java/com/example/datadelorean/serialization/UUIDSerializer.java
> Work well for me
>
>
>
> ----------------
> Кому: Zhanghao Chen (zhanghao.c...@outlook.com), Richard Cheung (
> rcheungsi...@gmail.com), user@flink.apache.org (user@flink.apache.org);
> Тема: Apache Flink Serialization Question;
> 15.05.2025, 15:56, "Schwalbe Matthias" <matthias.schwa...@viseca.ch>:
>
> Hi Richard,
>
>
>
> Same problem, 12 Flink versions later,
>
>
>
> I created my own TypeInformation/Serializer/Snapshot for UUID (Scala in
> that case), along:
>
>
>
> class UUIDTypeInformation extends TypeInformation[UUID]
>
> …
>
> class UUIDSerializer extends TupleSerializerBase[UUID](
>
> …
>
> class UUIDSerializerSnapshot(serializer: UUIDSerializer) extends
> CompositeTypeSerializerSnapshot[UUID, UUIDSerializer](serializer) {
>
> …
>
>
>
> Altogether 100 lines of code, implementation is simple, you only need to
> implement the respective apis and ‘follow the types’
>
> And the register the UUIDTypeInformation with the Flink type system.
>
>
>
>
>
> … unfortunately I haven’t got the permission to share the code …
>
>
>
> Hope that helps 😊
>
>
>
> This
>
>
>
>
>
>
>
>
>
>
>
>
>
> *From:* Zhanghao Chen <zhanghao.c...@outlook.com>
> *Sent:* Wednesday, May 14, 2025 3:00 AM
> *To:* Richard Cheung <rcheungsi...@gmail.com>; user@flink.apache.org
> *Subject:* [External] Re: Apache Flink Serialization Question
>
>
>
> ⚠*EXTERNAL MESSAGE – CAUTION: Think Before You Click *⚠
>
>
>
> Flink still use PojoSerializer for the class while only using Kryo for the
> UUID field.
>
>
>
> Best,
>
> Zhanghao Chen
> ------------------------------
>
> *From:* Richard Cheung <rcheungsi...@gmail.com>
> *Sent:* Wednesday, May 14, 2025 3:21
> *To:* user@flink.apache.org <user@flink.apache.org>
> *Subject:* Apache Flink Serialization Question
>
>
>
> Hi all!
>
>
>
> I have a question about serialization of POJO data classes in Apache Flink
> v1.8 (Java). Here's some context:
>
>
>
> One of my data classes was initially not POJO compatible as it had final
> fields and no no-arg constructor. I removed the final modifiers and added a
> no-arg constructor, and confirmed that the serializer switched from
> KryoSerializer to PojoSerializer using TypeInformation and TypeSerializer
> to log the info.
>
>
>
> However, one of the fields in the class is a java.util.UUID, which is not
> POJO compatible. I see a log message saying that the UUID field cannot be
> used as a POJO type. The logs also say that the UUID field will be
> processed as a generic type which means that, to my knowledge, the
> KryoSerializer will be used. These logs appear after the one indicating
> that the PojoSerializer is being used for my data class.
>
>
>
> My question is:
>
> Does the presence of the UUID field cause the entire class to be
> serialized with KryoSerializer or does Flink still use PojoSerializer for
> the class while only using Kryo for the UUID field?
>
>
>
> Best regards,
>
> Richard
> Diese Nachricht ist ausschliesslich für den Adressaten bestimmt und
> beinhaltet unter Umständen vertrauliche Mitteilungen. Da die
> Vertraulichkeit von e-Mail-Nachrichten nicht gewährleistet werden kann,
> übernehmen wir keine Haftung für die Gewährung der Vertraulichkeit und
> Unversehrtheit dieser Mitteilung. Bei irrtümlicher Zustellung bitten wir
> Sie um Benachrichtigung per e-Mail und um Löschung dieser Nachricht sowie
> eventueller Anhänge. Jegliche unberechtigte Verwendung oder Verbreitung
> dieser Informationen ist streng verboten.
>
> This message is intended only for the named recipient and may contain
> confidential or privileged information. As the confidentiality of email
> communication cannot be guaranteed, we do not accept any responsibility for
> the confidentiality and the intactness of this message. If you have
> received it in error, please advise the sender by return e-mail and delete
> this message and any attachments. Any unauthorised use or dissemination of
> this information is strictly prohibited.
>
>
