Re: Table API: Custom Kafka formats for Confluent Protobuf and JSON

Hey Yaroslav!, I suppose I will try it like this. The lookup would indeed be nice too, I will have a closer look at the corresponding source code. Thanks! -Theo > On 29. Nov 2022, at 17:41, Yaroslav Tkachenko wrote: > > Hey Theodor, > > That's pretty much it, assuming your Protobuf schema is

Re: Table API: Custom Kafka formats for Confluent Protobuf and JSON

Hey Theodor, That's pretty much it, assuming your Protobuf schema is more or less fixed. But for a production workload, you'd need to add a Schema Registry lookup. I guess the implementation for that would be similar to what's in the Avro format. On Tue, Nov 29, 2022 at 2:26 AM Theodor Wübker wr

Table API: Custom Kafka formats for Confluent Protobuf and JSON

Hey all, so Confluent has Kafka serializers to serialize Protobuf, Avro and JSON that create messages with a magic byte followed by a 4 byte schema id followed by the actual payload (refer the docs