Wouldn't this happen naturally? the large batches would just take a longer time to complete already.
On Thu, Jul 1, 2021 at 6:32 AM András Kolbert <kolbertand...@gmail.com> wrote: > Hi, > > I have a spark streaming application which generally able to process the > data within the given time frame. However, in certain hours it starts > increasing that causes a delay. > > In my scenario, the number of input records are not linearly increase the > processing time. Hence, ideally I'd like to increase the number of > batches/records that are being processed after a delay reaches a certain > time. > > Is there a possibility/settings to do so? > > Thanks > Andras > > > [image: image.png] >
