Hi, Kartik.
On flink ui, is there any operator that maintains a relatively high busy? Could 
you also try using a flame graph to provide more information?[1]


[1] 
https://nightlies.apache.org/flink/flink-docs-master/docs/ops/debugging/flame_graphs/




--

    Best!
    Xuyang




At 2024-03-30 18:07:23, "Kartik Kushwaha" <kushwaha.karti...@gmail.com> wrote:

Hello,


I have a Streaming event processing job that looks like this.


Source - ProcessFn(3 in total) - Sink


I am seeing a delay of 50ms to 250ms between each operators (maybe buffering or 
serde delays) leading to a slow end- to-end processing. What could be the 
reason for such high latency?


Some more details:
- Source operator is getting continuous events at a rate of 200 to 300 events 
per minute through controlled tests.
- Using DataStream<POJO> between the operators. It has simple types and the 
input payload got from source in byte[] format as fields. Right now the size of 
the payload is in few kb's. 
- Same events are also processed by another Flink job that looks 
source-processFn(mongoWriter)-sink. Here the end-to-end processing is less than 
5ms. Similar Stream<pojo> is being carried.
- The original(problematic) pipeline, has extraction, validation,  
transformation processFn. But each of these steps get completed within couple 
of ms. I am calculating the processing time inside these process functions by 
endTime - startTime logic in the java code. So the Average time of the event 
inside the operators is just 1ms. 
- There is no back pressure shown in the flink ui on these operators. 
- Input events are continously flowing from the source at  a very high rate 
without any delays. So waiting on the buffer can be ruled out.


Regards, 
Kartik











Reply via email to