Use Structured Streaming. Its aggregation, by definition, is across batches.
On Thu, Feb 27, 2020 at 3:17 PM Something Something < mailinglist...@gmail.com> wrote: > We've a Spark Streaming job that calculates some values in each batch. > What we need to do now is aggregate values across ALL batches. What is the > best strategy to do this in Spark Streaming. Should we use 'Spark > Accumulators' for this? >
