Hi all, Is anyone else looking into using Fury <https://github.com/alipay/fury> as a serializer for Flink?
I’m asking because the early results of my tests are very encouraging. Check out performance results here <https://gist.github.com/kkrugler/98c898797348d1eb5c5da60ee43d617f>. The meanings for each column are: Custom: annotate POJO class with TypeInfoFactory, and create hand-crafted serializer (so no use of Kryo, POJO, or other built-in Flink serialization) Fury: annotate POJO class with TypeInfoFactor, and create serializer based on Fury. Tuple: use Tuple instead of POJO. Row: use Row instead of POJO. Kryo+Fury: Use Kryo with Fury-based serializer for POJO. POJO: Use Flink POJO serializer. Kryo: Force Kryo to serialize POJO. Net-net is that Fury is almost 2x faster than the POJO serializer, but can handle generic types the same as Kryo. And it supposedly handles schema evolution, though I haven’t tried that, assuming you configure the builder properly. Currently they’re not guaranteeing binary compatibility across 0.x releases until 1.0 is out (currently at 0.4.1), so might make sense to wait for that. — Ken -------------------------- Ken Krugler http://www.scaleunlimited.com Custom big data solutions Flink & Pinot
