Update:
As the Pipeline progresses, the delay between the time when the batch is
completed and the time when onBatchCompleted() event is getting triggered
for the batch is increasing i.e. we are *not* observing a constant delay of
3 to 5 seconds for every batch.
Trimmed Logs which show the issue
Hi,
*Context*:
We are using the Spark Streaming Library.
We have created a StreamingListener to implement some logic when
onBatchCompleted() event is triggered. This StreamingListener is registered
with the StreamingContext.
We are using Spark on Kubernetes. The Spark version is 2.4.2. batchDura
