Hi all, I am trying to understand the situation with state serialization in flink. I'm looking at a number of sources, but slide 35 from here crystalizes my confusion:
https://www.slideshare.net/FlinkForward/flink-forward-berlin-2017-tzuli-gordon-tai-managing-state-in-apache-flink So, I understand that if 'Flink's own serialization stack' is unable to serialize a type you define, then it will fall back on Kryo generics. In this case, I believe what I'm being told is that state compatibility is difficult to ensure, and schema evolution in your jobs is not possible. However on this slide, they say " Kryo is generally not recommended ... Serialization frameworks with schema evolution support is recommended: Avro, Thrift " So is this implying that Flink's non-default serialization stack does not support schema evolution? In this case is it best practice to register custom serializers whenever possible. Thanks -- _Grab is hiring. Learn more at *https://grab.careers <https://grab.careers/>*_ By communicating with Grab Inc and/or its subsidiaries, associate companies and jointly controlled entities (“Grab Group”), you are deemed to have consented to processing of your personal data as set out in the Privacy Notice which can be viewed at https://grab.com/privacy/ <https://grab.com/privacy/> This email contains confidential information and is only for the intended recipient(s). If you are not the intended recipient(s), please do not disseminate, distribute or copy this email and notify Grab Group immediately if you have received this by mistake and delete this email from your system. Email transmission cannot be guaranteed to be secure or error-free as any information therein could be intercepted, corrupted, lost, destroyed, delayed or incomplete, or contain viruses. Grab Group do not accept liability for any errors or omissions in the contents of this email arises as a result of email transmission. All intellectual property rights in this email and attachments therein shall remain vested in Grab Group, unless otherwise provided by law.
