Hi Vishal, Kryo has a serializer called `CompatibleFieldSerializer` that allows for simple backward compatibility changes, such as adding non-optional fields / removing fields.
If using the KryoSerializer is a must, then a good thing to do is to register Kryo's `CompatibleFieldSerializer` as the serializer to be used for that specific type. By default, Kryo doesn’t use the `CompatibleFieldSerializer`, so you would have to explicitly register this. The issue is, I think it wouldn’t be possible to use the `CompatibleFieldSerializer` _after_ the bytes were already written with the default, non-compatible version (the `FieldSerializer`). So, AFAIK, this should only work if your Kryo state type was registered with the `CompatibleFieldSerializer` from the very beginning. There could be a workaround, but unfortunately that would require changing some code in the `KryoSerializer`. If you require this because your job is already running in production and data was already written by the `FieldSerializer`, please let me know and I’ll go into more details about this. Cheers, Gordon On 21 June 2018 at 10:14:15 AM, Fabian Hueske (fhue...@gmail.com) wrote: Hi Vishal, In general, Kryo serializers are not very upgrade friendly. Serializer compatibility [1] might be right approach here, but Gordon (in CC) might know more about this. Best, Fabian [1] https://ci.apache.org/projects/flink/flink-docs-release-1.5/dev/stream/state/custom_serialization.html#handling-serializer-upgrades-and-compatibility 2018-06-18 12:06 GMT+02:00 Vishal Santoshi <vishal.santo...@gmail.com>: Any more insight? On Wed, Jun 13, 2018, 3:34 PM Vishal Santoshi <vishal.santo...@gmail.com> wrote: Any ideas on the standard way ( or any roundabout way ) of doing a version upgrade that looks back ward compatible. The @FieldSerializer.Optional("0") actually does ignore the field ( even if reset ) giving it the default value if kyro is used. It has to do with the FieldSerializer behaves . There is another Serializer ( Composite I believe ) that allows for such back ward compatible changes. I know some work is being done in 1.6 to allow for above use case and I think Google Data Flow does provide some avenues. Thanks much Vishal On Tue, Jun 12, 2018 at 11:30 PM, Vishal Santoshi <vishal.santo...@gmail.com> wrote: I have a running pipe with Window State in a class say Class A{ long a; } It uses the default KryoSerializer I want to add a field to Class A { long a; long b; } I need to suspend with SP and resume with the new version of Class A Is there a definite way to do this. I tried Class A { long a; @FieldSerializer.Optional("0") long b; } but that seems to default to 0 , even when the Aggregation is putting in values. Could somebody give pointers as to how to solve this Thanks a ton.
