Hello guys, We're using Spark 3.5.0 for processing Kafka source that contains protobuf serialized data. The format is as follows:
message Request { long sent_ts = 1; Event[] event = 2; } message Event { string event_name = 1; bytes event_bytes = 2; } The event_bytes contains the data for the event_name. event_name is the className of the Protobuf class. Currently, we parse the protobuf message from the Kafka topic, and for every event in the array, push the event_bytes to the `event_name` topic, over which spark jobs run and use the same event_name protobuf class to deserialize the data. Is there a better way to do all this in a single job?