Hello Dan,
the technical reason is that this metric is only collected in the
*InputProcessor classes, which aren't used for source tasks.
I do recall that there were discussions about source watermarks, but
frankly i don't remember why we didn't add them.
In order to add them one would only have to modify the SourceContext
classes to a) store the last emitted watermark and b) expose it through
a metric.
Regards,
Chesnay
On 29.09.2016 23:03, dan bress wrote:
I am trying to measure the currentLowWatermark throughout my dataflow, but
I am not seeing it for tasks with sources. For those tasks I see these
metrics:
lastCheckpointSize
numBytesInLocal
numBytesInRemote
numBytesOut
why am I not seeing currentLowWatermark on these tasks?
Thanks!
Dan
