Thank you for details.
I expect that StartBundle and FinishBundle methods are called in your DoFn
(KafkaMessageExtractor). Since it’s called before your Window transform, I
don’t think it should affect it.
Did you count how many records are processed by one bundle?
What is actually happening
Hi,
I check the time when StartBundle is called and do the same thing for
FinishBundle then take the difference between Start and Finish Bundle times
and report bundle latency. I put this metric on
a step(KafkaMessageExtractor) which is right after the KafkaIO step. I dont
know if this is related,
Hi Talat,
Could you elaborate what do you mean by “opening and closing bundle”?
Sometimes, starting a KafkaReader can take time since it will seek for a start
offset for each assigned partition but it happens only once at pipeline
start-up and mostly depends on network conditions.
> On 9 Jun 2
