Ok thanks Michael for all your help!
--
Dhruv Kumar
PhD Candidate
Department of Computer Science and Engineering
University of Minnesota
www.dhruvkumar.me
> On Apr 26, 2018, at 19:24, TechnoMage wrote:
>
> Yes, Kafka for source and sink which make
Yes, Kafka for source and sink which makes monitoring the Flink in/out easy.
Michael
> On Apr 26, 2018, at 5:27 PM, Dhruv Kumar wrote:
>
> Ok that answers my questions.
>
> What are you keeping the source and sink as? Is it Kafka for both?
>
> -
Ok that answers my questions.
What are you keeping the source and sink as? Is it Kafka for both?
--
Dhruv Kumar
PhD Candidate
Department of Computer Science and Engineering
University of Minnesota
www.dhruvkumar.me
> On Apr 26, 2018, at 16:37, Tech
Yes NTP can still have skew. It may be measured in fractions of a second, but
with Flink that can be significant if you care about sub-second latency
accuracy. Since I have a 20 stage stream with 0.002 second latency it can
matter.
Back pressure is the limiting of input due to the inability o
What do you mean by the time skew from one machine(source) to another(sink)? Do
you mean the system time clocks of the source and sink may not be in sync. If I
regularly use NTP to keep the system clocks in sync, will time skew still
happen?
Could you also elaborate on what do you mean by back
In a single machine system this may work ok. In a multi-machine system this is
not as reliable as the time skew from one machine (source) to another (sink)
can impact the measurements. This also does not account for back presure on
the source. We are using an external process to in parallel r
Hi
I was trying to compute the end-to-end-latency for each record processed by
Flink. By end-to-end latency, I mean the difference between the time at which
the record entered the Flink system (came at source) and the time at which the
record is finally emitted into the sink. What is the best w
