Hi guys, I am new to flink framework. And we are building an application that takes kinesis stream for both flink source and sink. The flink version we are using is 1.4.2, which is also the version for the flink-connector-kinesis. We built the flink-connector-kinesis jar explicitly with KPL version 0.12.6 due to the existing problems with default 0.12.5.
I get a rough idea how the backpressure works with flink through reading http://mail-archives.apache.org/mod_mbox/flink-user/201801.mbox/%3cf8dd76c0-9de0-412a-8c24-b72af0d42...@data-artisans.com%3E From my experiment with flink and flink-connector-kinesis, the back pressure only happens within flink processing operations, i.e., not in the flink producer to kinesis. More specifically, when the throughput from KPL exceeds the kinesis throughput limitations, flink does not slow down at all, i.e., it does not add pressure on the processing chain up to the flink consumer. Correct me if I misunderstood this. It looks like the flink producer (in the flink-connector-kinesis) is a standalone component, once a record is collected and sent to the producer, flink core finishes all the processing and does not care the fate of the record any more, it is the responsibility of the connector to continue the job. I am expecting back pressure to happen from the source kinesis stream to the sink kinesis stream, whenever the sink kinesis stream could not handle the volume, it adds back pressure. Could someone illustrate a bit more why flink connector is designed in such a way. Also correct me if I stated anything wrong. Gavin Liu
