How will i can to know that for how much time particular RDD had
remained in pipeline .
On Fri, Jun 19, 2015 at 7:59 AM, Tathagata Das wrote:
> Why do you need to uniquely identify the message? All you need is the time
> when the message was inserted by the receiver, and when it is processed,
Why do you need to uniquely identify the message? All you need is the time
when the message was inserted by the receiver, and when it is processed,
isnt it?
On Thu, Jun 18, 2015 at 2:28 PM, anshu shukla
wrote:
> Thanks alot , But i have already tried the second way ,Problem with that
> is tha
Thanks alot , But i have already tried the second way ,Problem with that
is that how to identify the particular RDD from source to sink (as we can
do by passing a msg id in storm) . For that i just updated RDD and added
a msgID (as static variable) . but while dumping them to file some of the
Couple of ways.
1. Easy but approx way: Find scheduling delay and processing time using
StreamingListener interface, and then calculate "end-to-end delay = 0.5 *
batch interval + scheduling delay + processing time". The 0.5 * batch
inteval is the approx average batching delay across all the record
Sorry , i missed the LATENCY word.. for a large streaming query .How to
find the time taken by the particular RDD to travel from initial
D-STREAM to final/last D-STREAM .
Help Please !!
On Fri, Jun 19, 2015 at 12:40 AM, Tathagata Das wrote:
> Its not clear what you are asking. Find "what
Its not clear what you are asking. Find "what" among RDD?
On Thu, Jun 18, 2015 at 11:24 AM, anshu shukla
wrote:
> Is there any fixed way to find among RDD in stream processing systems ,
> in the Distributed set-up .
>
> --
> Thanks & Regards,
> Anshu Shukla
>
Is there any fixed way to find among RDD in stream processing systems ,
in the Distributed set-up .
--
Thanks & Regards,
Anshu Shukla
