Hi devs, Thanks for all the feedback. If there are no more comments, I would like to start a vote for this FLIP, thanks again!
Best, Fang Yong On Wed, Dec 20, 2023 at 9:12 PM Yong Fang <zjur...@gmail.com> wrote: > Hi Ken, > > Thanks for your feedback. The purpose of this FLIP is to improve the use > of serialization, including configurable serializer for users, providing > serializer for composite data types, and resolving the default enabling of > Kryo, etc. Introducing a better serialization framework would be a great > help for Flink's performance, and it's great to see your tests on Fury. > However, as @Xintong mentioned, this could be a huge work and beyond the > scope of this FLIP. If you're interested, I think we could create a new > FLIP for it and discuss it further. What do you think? Thanks. > > Best, > Fang Yong > > On Mon, Dec 18, 2023 at 11:16 AM Xintong Song <tonysong...@gmail.com> > wrote: > >> Hi Ken, >> >> I think the main purpose of this FLIP is to change how users interact with >> the knobs for customizing the serialization behaviors, from requiring code >> changes to working with pure configurations. Redesigning the knobs (i.e., >> names, semantics, etc.), on the other hand, is not the purpose of this >> FLIP. Preserving the existing names and semantics should also help >> minimize >> the migration cost for existing users. Therefore, I'm in favor of not >> changing them. >> >> Concerning decoupling from Kryo, and introducing other serialization >> frameworks like Fury, I think that's a bigger topic that is worth further >> discussion. At the moment, I'm not aware of any community consensus on >> doing so. And even if in the future we decide to do so, the changes needed >> should be the same w/ or w/o this FLIP. So I'd suggest not to block this >> FLIP on these issues. >> >> WDYT? >> >> Best, >> >> Xintong >> >> >> >> On Fri, Dec 15, 2023 at 1:40 AM Ken Krugler <kkrugler_li...@transpac.com> >> wrote: >> >> > Hi Yong, >> > >> > Looks good, thanks for creating this. >> > >> > One comment - related to my recent email about Fury, I would love to see >> > the v2 serialization decoupled from Kryo. >> > >> > As part of that, instead of using xxxKryo in methods, call them >> xxxGeneric. >> > >> > A more extreme change would be to totally rely on Fury (so no more POJO >> > serializer). Fury is faster than the POJO serializer in my tests, but >> this >> > would be a much bigger change. >> > >> > Though it could dramatically simplify the Flink serialization support. >> > >> > — Ken >> > >> > PS - a separate issue is how to migrate state from Kryo to something >> like >> > Fury, which supports schema evolution. I think this might be possible, >> by >> > having a smarter deserializer that identifies state as being created by >> > Kryo, and using (shaded) Kryo to deserialize, while still writing as >> Fury. >> > >> > > On Dec 6, 2023, at 6:35 PM, Yong Fang <zjur...@gmail.com> wrote: >> > > >> > > Hi devs, >> > > >> > > I'd like to start a discussion about FLIP-398: Improve Serialization >> > > Configuration And Usage In Flink [1]. >> > > >> > > Currently, users can register custom data types and serializers in >> Flink >> > > jobs through various methods, including registration in code, >> > > configuration, and annotations. These lead to difficulties in >> upgrading >> > > Flink jobs and priority issues. >> > > >> > > In flink-2.0 we would like to manage job data types and serializers >> > through >> > > configurations. This FLIP will introduce a unified option for data >> type >> > and >> > > serializer and users can configure all custom data types and >> > > pojo/kryo/custom serializers. In addition, this FLIP will add more >> > built-in >> > > serializers for complex data types such as List and Map, and optimize >> the >> > > management of Avro Serializers. >> > > >> > > Looking forward to hearing from you, thanks! >> > > >> > > [1] >> > > >> > >> https://cwiki.apache.org/confluence/display/FLINK/FLIP-398%3A+Improve+Serialization+Configuration+And+Usage+In+Flink >> > > >> > > Best, >> > > Fang Yong >> > >> > -------------------------- >> > Ken Krugler >> > http://www.scaleunlimited.com >> > Custom big data solutions >> > Flink & Pinot >> > >> > >> > >> > >> >
