Maybe add another stat for batches waiting in the job queue ? Cheers
On Fri, Apr 3, 2015 at 10:01 AM, Tathagata Das <t...@databricks.com> wrote: > Very good question! This is because the current code is written such that > the ui considers a batch as waiting only when it has actually started being > processed. Thats batched waiting in the job queue is not considered in the > calculation. It is arguable that it may be more intuitive to count that in > the waiting as well. > On Apr 3, 2015 12:59 AM, "bit1...@163.com" <bit1...@163.com> wrote: > >> >> I copied the following from the spark streaming UI, I don't know why the >> Waiting batches is 1, my understanding is that it should be 72. >> Following is my understanding: >> 1. Total time is 1minute 35 seconds=95 seconds >> 2. Batch interval is 1 second, so, 95 batches are generated in 95 seconds. >> 3. Processed batches are 23(Correct, because in my processing code, it >> does nothing but sleep 4 seconds) >> 4. Then the waiting batches should be 95-23=72 >> >> >> >> - *Started at: * Fri Apr 03 15:17:47 CST 2015 >> - *Time since start: *1 minute 35 seconds >> - *Network receivers: *1 >> - *Batch interval: *1 second >> - *Processed batches: *23 >> - *Waiting batches: *1 >> - *Received records: *0 >> - *Processed records: *0 >> >> >> ------------------------------ >> bit1...@163.com >> >
