Structured Streaming does not maintain a queue of batch like DStream. DStreams used to cut off batches at a fixed interval and put in a queue, and a different thread processed queued batches. In contrast, Structured Streaming simply cuts off and immediately processes a batch after the previous batch finishes. So the question about queue size and lag does not apply to Structured Streaming.
That said, there is no UI for Structured Streaming. You can see the sql plans for each micro-batch in the SQL tab. On Wed, Jun 20, 2018 at 12:12 PM, SRK <swethakasire...@gmail.com> wrote: > hi, > > How do we get information like lag and queued up batches in Structured > streaming? Following api does not seem to give any info about lag and > queued up batches similar to DStreams. > > https://spark.apache.org/docs/2.2.1/api/java/org/apache/ > spark/streaming/scheduler/BatchInfo.html > > Thanks! > > > > > -- > Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/ > > --------------------------------------------------------------------- > To unsubscribe e-mail: user-unsubscr...@spark.apache.org > >
