Re: Converting spark batch to spark streaming

Hi, Start with DataStreamWriter.foreachBatch. Pozdrawiam, Jacek Laskowski https://about.me/JacekLaskowski "The Internals Of" Online Books Follow me on https://twitter.com/jaceklaskowski On Thu, Jan 7, 2021 at 6:55 PM mhd wrk

Converting spark batch to spark streaming

I'm trying to convert a spark batch application to a streaming application and wondering what function (or design pattern) I should use to execute a series of operations inside the driver upon arrival of each message (a text file inside an HDFS folder) before starting computation inside executors.