Can someone please give input for below?
From: Kamal Mittal <[email protected]>
Sent: 22 September 2025 17:16
To: Kamal Mittal <[email protected]>; [email protected]
Subject: RE: Flink kafka source with Avro Generic Record
Hello,
I tried this and Flink fails later, when it tries to serialize/deserialize the
GenericRecord object for communication between operators (e.g. from map() to
another operator, or writing checkpoints, or shuffling).
it's a serialization issue during operator chaining or data exchange in Flink’s
runtime.
Probable reason:
GenericRecord from Avro holds schema metadata internally, which includes
unmodifiable maps, especially:
schema (org.apache.avro.generic.GenericData$Record)
↳ fieldMap (org.apache.avro.Schema$RecordSchema)
↳ reserved (java.util.Collections$UnmodifiableMap)
These types (like UnmodifiableMap) are not easily serializable by Kryo, which
Flink falls back to if:
* No proper TypeInformation or TypeSerializer is provided.
* Flink cannot infer a more optimized serializer.
Error Stack :
Caused by: com.esotericsoftware.kryo.KryoException:
java.lang.UnsupportedOperationException
Serialization trace:
reserved (org.apache.avro.Schema$Field)
fieldMap (org.apache.avro.Schema$RecordSchema)
schema (org.apache.avro.generic.GenericData$Record)
Can you please confirm above understanding and also possible way to resolve
this? Probably custom serializer and custom type information solution needed
here, is that recommended?
Rgds,
Kamal
From: Kamal Mittal via user
<[email protected]<mailto:[email protected]>>
Sent: 22 September 2025 13:56
To: [email protected]<mailto:[email protected]>
Subject: Flink kafka source with Avro Generic Record
Hello,
I need to support Flink application accepting avro binary events with different
schemas over flink kafka source. Need to use custom schema registry to fetch
schema at runtime and decode the incoming event.
Will use Avro Generic Record to decode incoming event with different avro
schemas.
Gone through the page - Flink Serialization Tuning Vol. 1: Choosing your
Serializer — if you can | Apache
Flink<https://flink.apache.org/2020/04/15/flink-serialization-tuning-vol.-1-choosing-your-serializer-if-you-can/#avro-generic>.
Can you please tell as at compile/job graph time schema is not available then
it will use Kryo as serialzer? Also anything can be done here to improve it as
for Kryo perf. is impacted?
Rgds,
Kamal
