Hello, Can someone please assist for this query? Thanks!
Thanks, Sandeep > On 06-May-2021, at 10:30 AM, Sandeep khanzode <sand...@shiftright.ai> wrote: > > Hi, > > Is there a working example somewhere that I can refer for writing Avro > entities in Flink state as well as Avro serializaition in > KafkaConsumer/Producer? > > I tried to use Avro entities directly but there is an issue beyond Apache > Avro 1.7.7 in that the entities created have a serialVersionUid. So when I > tried to test schema evolution by adding a member, there was the java > serialization issue saying the two generated classes’ serialVersionUids do > not match i.e. the one stored in the state and the one being used with the > new member variable now. > > Is there any configuration that overrides this? > > Request you to please provide some references of samples. Thanks. > > Thanks, > Sandeep > > >> On 30-Apr-2021, at 5:00 PM, Timo Walther <twal...@apache.org> wrote: >> >> I also found these pages: >> >> https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/state/schema_evolution.html >> >> https://flink.apache.org/news/2020/04/15/flink-serialization-tuning-vol-1.html#avro >> >> I hope this helps. >> >> Regards, >> Timo >> >> >> On 30.04.21 13:20, Sandeep khanzode wrote: >>> Hi Timo, >>> Thanks! I will take a look at the links. >>> Can you please share if you have any simple (or complex) example of Avro >>> state data structures? >>> Thanks, >>> Sandeep >>>> On 30-Apr-2021, at 4:46 PM, Timo Walther <twal...@apache.org> wrote: >>>> >>>> Hi Sandeep, >>>> >>>> did you have a chance to look at this documentation page? >>>> >>>> https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/state/custom_serialization.html >>>> >>>> The interfaces might not be easy to implement but are very powerful to >>>> address compatibility issues. You can also look into Flink serializers for >>>> some examples: >>>> >>>> https://github.com/apache/flink/tree/master/flink-core/src/main/java/org/apache/flink/api/java/typeutils/runtime >>>> >>>> Esp: >>>> >>>> https://github.com/apache/flink/blob/89c6c03660a88a648bbd13b4e6696124fe46d013/flink-core/src/main/java/org/apache/flink/api/java/typeutils/runtime/PojoSerializer.java#L599 >>>> >>>> For the POJO logic. >>>> >>>> By the way, usually we recommend Avro for state data structures if schema >>>> evolution is a topic. >>>> >>>> Regards, >>>> Timo >>>> >>>> >>>> >>>> On 29.04.21 18:10, Sandeep khanzode wrote: >>>>> Hello, >>>>> Is there a working example of a TypeSerializer for a Java type stored in >>>>> the State? >>>>> My requirement is that I should be able to store the Java POJO entity in >>>>> the MapState. The state is backed by RocksDBBackend. >>>>> If I update the entity with a new member variable, I am unable to >>>>> deserialise the state into the new entity. >>>>> I checked this link. >>>>> https://ci.apache.org/projects/flink/flink-docs-release-1.12/dev/types_serialization.html >>>>> >>>>> <https://ci.apache.org/projects/flink/flink-docs-release-1.12/dev/types_serialization.html> >>>>> It does mention that the POJO type is special based on the rules. Does >>>>> that mean that I can add or remove member variables for the POJO? I have >>>>> been unable to get it to work. >>>>> Thanks, >>>>> Sandeep >>>> >> >
