HI , I am trying to do some performance test to my flink deployment. I am implementing an extremely simplistic use case I built a ZMQ Source
The topology is ZMQ Source - > Mapper- > DIscardingSInk ( a sink that does nothing ) Data is pushed via ZMQ at a very high rate. When the incoming rate from ZMQ is higher then the rate flink can keep up with, I can see that the JVM Heap is filling up ( using Flinks metrics ) . when the heap is fullt consumes – TM chokes , a HeartBeat is missed and the job fails. I was expecting Flink to handle this type of backpressure gracefully and not Note : The mapper has not state to persist See below the Grafana charts, on the left is the TM HHEAP Used. On the right – the ZMQ – out of flink. ( yellow ) Vs Flink consuming rate from reported by ZMQSOurce 1GB is the configured heap size [cid:image001.jpg@01D5A422.7538D5C0]
