Actually, yes. I have a job already running with "FieldSerializer" in production. Any insights will be appreciated.
On Sat, Jun 23, 2018 at 7:39 AM, Vishal Santoshi <vishal.santo...@gmail.com> wrote: > Thanks. > > On Thu, Jun 21, 2018 at 4:34 AM, Tzu-Li (Gordon) Tai <tzuli...@apache.org> > wrote: > >> Hi Vishal, >> >> Kryo has a serializer called `CompatibleFieldSerializer` that allows for >> simple backward compatibility changes, such as adding non-optional fields / >> removing fields. >> >> If using the KryoSerializer is a must, then a good thing to do is to >> register Kryo's `CompatibleFieldSerializer` as the serializer to be used >> for that specific type. >> By default, Kryo doesn’t use the `CompatibleFieldSerializer`, so you >> would have to explicitly register this. >> >> The issue is, I think it wouldn’t be possible to use the >> `CompatibleFieldSerializer` _after_ the bytes were already written with the >> default, non-compatible version (the `FieldSerializer`). >> So, AFAIK, this should only work if your Kryo state type was registered >> with the `CompatibleFieldSerializer` from the very beginning. >> There could be a workaround, but unfortunately that would require >> changing some code in the `KryoSerializer`. >> If you require this because your job is already running in production and >> data was already written by the `FieldSerializer`, please let me know and >> I’ll go into more details about this. >> >> Cheers, >> Gordon >> >> On 21 June 2018 at 10:14:15 AM, Fabian Hueske (fhue...@gmail.com) wrote: >> >> Hi Vishal, >> >> In general, Kryo serializers are not very upgrade friendly. >> Serializer compatibility [1] might be right approach here, but Gordon (in >> CC) might know more about this. >> >> Best, Fabian >> >> [1] https://ci.apache.org/projects/flink/flink-docs-release-1.5/ >> dev/stream/state/custom_serialization.html#handling- >> serializer-upgrades-and-compatibility >> >> 2018-06-18 12:06 GMT+02:00 Vishal Santoshi <vishal.santo...@gmail.com>: >> >>> Any more insight? >>> >>> On Wed, Jun 13, 2018, 3:34 PM Vishal Santoshi <vishal.santo...@gmail.com> >>> wrote: >>> >>>> Any ideas on the standard way ( or any roundabout way ) of doing a >>>> version upgrade that looks back ward compatible. >>>> The @FieldSerializer.Optional("0") actually does ignore the field ( >>>> even if reset ) giving it the default value if kyro is used. It has to do >>>> with the FieldSerializer behaves . There is another Serializer ( >>>> Composite I believe ) that allows for such back ward compatible changes. >>>> >>>> >>>> I know some work is being done in 1.6 to allow for above use case and I >>>> think Google Data Flow does provide some avenues. >>>> >>>> Thanks much >>>> >>>> Vishal >>>> >>>> >>>> >>>> On Tue, Jun 12, 2018 at 11:30 PM, Vishal Santoshi < >>>> vishal.santo...@gmail.com> wrote: >>>> >>>>> I have a running pipe with Window State in a class say >>>>> >>>>> Class A{ >>>>> long a; >>>>> } >>>>> >>>>> It uses the default KryoSerializer >>>>> >>>>> I want to add a field to >>>>> >>>>> Class A { >>>>> long a; >>>>> long b; >>>>> } >>>>> >>>>> I need to suspend with SP and resume with the new version of Class A >>>>> >>>>> >>>>> Is there a definite way to do this. I tried >>>>> >>>>> Class A { >>>>> long a; >>>>> @FieldSerializer.Optional("0") >>>>> long b; >>>>> } >>>>> >>>>> but that seems to default to 0 , even when the Aggregation is putting >>>>> in values. >>>>> >>>>> Could somebody give pointers as to how to solve this >>>>> >>>>> Thanks a ton. >>>>> >>>>> >>>>> >>>>> >>>> >> >
