Hi, I am using Structured Streaming on Azure HdInsight. The version is 2.4.6.
I am trying to understand the microbatch mode - default and fixed intervals. Does the fixed interval microbatch follow something similar to receiver based model where records keep getting pulled and stored into blocks for the duration of the interval at the end of which a job is kicked off? Or, does the job just process the current microbatch and sleep for the rest of the interval and pulls records only at the end of the interval? I am fully aware of the two dstreams models - receiver and direct based dstreams. I am just trying to figure out if either of these two models were reused in Structured Streaming. Regards, Sheel
