Hi Ufuk,
I have increased the sampling size to 1000 and decreased the refresh interval
by half. In my Kafka topic I have pumped million messages which is read by
KafkaConsumer pipeline and then pass it to a transofmation step where I have
introduced sleep (3 sec) for every single message received and the final step
is HDFS sink using RollingSinc API.
jobmanager.web.backpressure.num-samples: 1000
jobmanager.web.backpressure.refresh-interval: 30000
I was hoping to see the backpressure tab from UI to display some warning but I
still see "OK" message.
This makes me wonder if I am testing the backpressure scenario properly or not?
RegardsVijay
On Monday, March 7, 2016 3:19 PM, Ufuk Celebi <u...@apache.org> wrote:
Hey Vijay!
On Mon, Mar 7, 2016 at 8:42 PM, Vijay Srinivasaraghavan
<vijikar...@yahoo.com> wrote:
> 3) How can I simulate and verify backpressure? I have introduced some delay
> (Thread Sleep) in the job before the sink but the "backpressure" tab from UI
> does not show any indication of whether backpressure is working or not.
If a task is slow, it is back pressuring upstream tasks, e.g. if your
transformations have the sleep, the sources should be back pressured.
It can happen that even with the sleep the tasks still produce their
data as fast as they can and hence no back pressure is indicated in
the web interface. You can increase the sleep to check this.
The mechanism used to determine back pressure is based on sampling the
stack traces of running tasks. You can increase the number of samples
and/or decrease the delay between samples via config parameters shown
in [1]. It can happen that the samples miss the back pressure
indicators, but usually the defaults work fine.
[1]
https://ci.apache.org/projects/flink/flink-docs-master/setup/config.html#jobmanager-web-frontend
