The SQL plan of each micro-batch in the Spark UI (SQL tab) has links to the actual Spark jobs that ran in the micro-batch. From that you can drill down into the stage information. I agree that its not there as a nice per-stream table as with the Streaming tab, but all the information is present if you dig for it.
On Wed, Jun 27, 2018 at 4:14 PM, swetha kasireddy <swethakasire...@gmail.com > wrote: > Thanks TD, but the sql plan does not seem to provide any information on > which stage is taking longer time or to identify any bottlenecks about > various stages. Spark kafka Direct used to provide information about > various stages in a micro batch and the time taken by each stage. Is there > a way to find out stage level information like time take by each stage, > shuffle read/write data etc? Do you have any documentation on how to use > SQL tab for troubleshooting? > > On Wed, Jun 20, 2018 at 6:07 PM, Tathagata Das < > tathagata.das1...@gmail.com> wrote: > >> Also, you can get information about the last progress made (input rates, >> etc.) from StreamingQuery.lastProgress, StreamingQuery.recentProgress, and >> using StreamingQueryListener. >> Its all documented - https://spark.apache.org/docs/ >> latest/structured-streaming-programming-guide.html#monitorin >> g-streaming-queries >> >> On Wed, Jun 20, 2018 at 6:06 PM, Tathagata Das < >> tathagata.das1...@gmail.com> wrote: >> >>> Structured Streaming does not maintain a queue of batch like DStream. >>> DStreams used to cut off batches at a fixed interval and put in a queue, >>> and a different thread processed queued batches. In contrast, Structured >>> Streaming simply cuts off and immediately processes a batch after the >>> previous batch finishes. So the question about queue size and lag does not >>> apply to Structured Streaming. >>> >>> That said, there is no UI for Structured Streaming. You can see the sql >>> plans for each micro-batch in the SQL tab. >>> >>> >>> >>> >>> >>> On Wed, Jun 20, 2018 at 12:12 PM, SRK <swethakasire...@gmail.com> wrote: >>> >>>> hi, >>>> >>>> How do we get information like lag and queued up batches in Structured >>>> streaming? Following api does not seem to give any info about lag and >>>> queued up batches similar to DStreams. >>>> >>>> https://spark.apache.org/docs/2.2.1/api/java/org/apache/spar >>>> k/streaming/scheduler/BatchInfo.html >>>> >>>> Thanks! >>>> >>>> >>>> >>>> >>>> -- >>>> Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/ >>>> >>>> --------------------------------------------------------------------- >>>> To unsubscribe e-mail: user-unsubscr...@spark.apache.org >>>> >>>> >>> >> >
