Writing Custom Spark Readers and Writers

Hey team, Can you please share some documentation/blogs where we can get to know how we can write custom sources and sinks for both streaming and static datasets. Thanks in advance Dyanesh Varun

protobuf data as input to spark streaming

Hello Experts Has anyone used protobuf (proto3) encoded data (from kafka) as input source and been able to do spark structured streaming? I would appreciate if you can share any sample code/example Regards Kiran >

Re: loop of spark jobs leads to increase in memory on worker nodes and eventually faillure

Hi, can you please give details around: spark version, what is the operation that you are running, why in loops, and whether you are caching in any data or not, and whether you are referencing the variables to create them like in the following expression we are referencing x to create x, x = x + 1