Re: Apache Flink - Reading Avro messages from Kafka with schema in schema registry

2021-07-15 Thread M Singh
Thanks so much Dawid for your time and explanations.  Mans On Thursday, July 15, 2021, 10:09:23 AM EDT, Dawid Wysakowicz wrote: Ad 1/2/4 Please refer to Avro's documentation on how do the reader and writer schemas work e.g. here[1] and here[2]. That's purely Avro's thing. Ad.3 Theo

Re: Apache Flink - Reading Avro messages from Kafka with schema in schema registry

2021-07-15 Thread Dawid Wysakowicz
Ad 1/2/4 Please refer to Avro's documentation on how do the reader and writer schemas work e.g. here[1] and here[2]. That's purely Avro's thing. Ad.3 Theoretically yes. The problem is the communication between TaskManagers. We need to serialize the Avro records somehow. The chosen approach is to u

Re: Apache Flink - Reading Avro messages from Kafka with schema in schema registry

2021-07-15 Thread M Singh
Hello Dawid: Thanks for your answers and references. I do have a few questions: 1. Is there any scenario where the reader and writer schema should differ ?  2. How is the mismatch b/w the two schemas (one passed as argument and other retrieved from schema registry) resolved at run time ? 3. As me

Re: Apache Flink - Reading Avro messages from Kafka with schema in schema registry

2021-07-13 Thread Dawid Wysakowicz
Hi, Yes, you are right the schema in the forGeneric is the readerSchema and at the same time the schema that Flink will be working with in the pipeline. It will be the schema used to serialize and deserialize records between different TaskManagers. Between the Flink TaskManagers that schema plays

Apache Flink - Reading Avro messages from Kafka with schema in schema registry

2021-07-08 Thread M Singh
Hi: I am trying to read avro encoded messages from Kafka with schema registered in schema registry. I am using the class (https://ci.apache.org/projects/flink/flink-docs-master/api/java/org/apache/flink/formats/avro/registry/confluent/ConfluentRegistryAvroDeserializationSchema.html) using the me