Additional metadata available for Kafka serdes

2024-03-12 Thread Balint Bene
Hello! Looking to get some guidance for a problem around the Flink formats used for Kafka. Flink currently uses common serdes interfaces across all formats. However, some data formats used in Kafka require headers for serdes. It's the same problem for serialization and deserialization, so I'll ju

Re: Additional metadata available for Kafka serdes

2024-03-14 Thread Balint Bene
gt; close. > > We did consider trying to use the Apicurio deser libraries but this is > tricky due to the way the code is split. > > Let me know what you think – I hope this approach will meet your needs, > Kind regards, David. > > From: Balint Bene > Date: Tuesday,

Re: [DISCUSS] FLIP-XXX Apicurio-avro format

2024-04-02 Thread Balint Bene
+1 to this as well, as we were talking about this on the other generic thread about supporting headers at the serdes layer. Couple of tiny points: > passHeadersToSerDe option If the interface can be made backwards compatible, is there a need for this option anymore? For example adding default fun